Nov 02 22:04:50.046916 ip-10-0-143-75 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:04:50.493240 ip-10-0-143-75 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:04:50.493240 ip-10-0-143-75 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:04:50.493240 ip-10-0-143-75 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:04:50.493240 ip-10-0-143-75 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:04:50.497482 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497358 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.497482 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497482 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.497482 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497486 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.497482 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497489 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497492 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497495 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497499 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497502 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497505 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497508 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497511 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497514 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497517 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497519 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497522 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497524 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497533 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497536 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497539 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497542 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497545 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497547 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.497624 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497550 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497553 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497556 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497558 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497561 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497577 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497580 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497582 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497585 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497588 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497590 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497593 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497595 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497598 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497601 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497605 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497609 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497615 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497618 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.498115 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497621 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497624 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497627 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497630 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497633 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497635 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497638 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497641 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497644 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497647 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497650 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497652 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497655 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497657 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497660 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497663 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497666 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497668 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497671 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497673 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.498556 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497676 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497680 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497683 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497687 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497690 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497693 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497696 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497699 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497703 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497706 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497709 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497712 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497714 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497716 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497719 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497721 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497724 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497727 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497730 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.499051 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497733 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497736 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497739 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497741 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497744 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497747 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497750 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497752 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497755 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497757 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497760 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497762 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497765 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497768 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497770 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497773 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497775 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497778 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497780 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497783 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.499510 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497786 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497789 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497792 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497794 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497797 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.497799 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498715 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498724 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498727 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498735 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498738 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498741 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498744 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498747 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498749 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498758 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498761 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498764 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498766 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.500008 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498769 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498771 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498773 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498776 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498779 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498783 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498787 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498790 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498793 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498795 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498798 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498800 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498803 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498807 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498811 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498813 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498816 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498818 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498821 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498824 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.500511 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498826 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498829 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498831 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498834 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498837 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498839 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498842 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498844 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498846 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498856 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498860 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498863 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498866 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498869 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498872 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498874 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498877 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498879 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498881 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.501019 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498884 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498886 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498889 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498891 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498894 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498898 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498901 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498904 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498908 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498910 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498913 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498915 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498918 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498921 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498923 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498926 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498928 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498931 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498933 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498936 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.501483 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498938 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498941 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498943 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498946 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498949 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498951 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498954 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498956 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498959 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498961 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498964 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498966 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498969 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498972 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498974 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498977 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498979 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498983 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498986 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498989 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.501966 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498991 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498995 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.498997 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499000 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499002 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499005 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499008 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499010 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499013 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499015 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499018 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499021 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499023 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499026 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499590 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499601 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499604 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499607 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499610 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.502439 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499613 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499615 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499617 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499620 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499623 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499625 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499628 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499630 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499633 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499636 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499639 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499643 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499645 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499648 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499650 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499653 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499655 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499658 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499660 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.502919 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499663 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499665 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499668 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499670 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499673 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499676 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499678 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499681 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499683 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499685 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499693 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499696 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499699 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499701 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499703 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499706 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499710 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499714 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499717 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.503378 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499720 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499722 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499725 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499727 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499730 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499733 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499736 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499738 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499741 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499743 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499746 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499748 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499751 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499753 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499756 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499758 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499760 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499763 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499765 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499768 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.503859 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499770 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499773 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499775 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499778 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499786 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499789 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499791 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499794 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499796 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499799 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499801 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499804 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499806 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499809 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499811 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499814 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499817 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499822 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499825 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499827 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.504328 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499830 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499832 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499835 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499839 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499842 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499845 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499853 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499856 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499859 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499861 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499863 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499866 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499868 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499871 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499873 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499876 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499878 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499886 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499889 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.504825 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499892 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499894 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499897 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.499899 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500845 2574 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500856 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500863 2574 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500868 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500873 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500877 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500882 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500886 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500890 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500892 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500896 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500899 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500902 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500905 2574 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500908 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500911 2574 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500914 2574 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500917 2574 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500926 2574 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500929 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:04:50.505309 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500932 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500935 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500938 2574 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500942 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500945 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500949 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500952 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500962 2574 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500966 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500969 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500972 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500975 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500979 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500982 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500985 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500988 2574 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500991 2574 flags.go:64] FLAG: --enable-server="true" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.500996 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501003 2574 flags.go:64] FLAG: --event-burst="100" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501006 2574 flags.go:64] FLAG: --event-qps="50" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501009 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501012 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501015 2574 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501020 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:04:50.505887 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501023 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501026 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501030 2574 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501033 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501036 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501039 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501042 2574 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501045 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501048 2574 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501051 2574 flags.go:64] FLAG: --feature-gates="" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501055 2574 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501058 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501061 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501064 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501067 2574 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501070 2574 flags.go:64] FLAG: --help="false" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501079 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501083 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501085 2574 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501088 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501092 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501095 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501098 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:04:50.506467 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501101 2574 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501104 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501108 2574 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501111 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501114 2574 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501117 2574 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501120 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501123 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501126 2574 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501129 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501132 2574 flags.go:64] FLAG: --lock-file="" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501135 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501138 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501141 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501146 2574 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501149 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501152 2574 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501155 2574 flags.go:64] FLAG: --logging-format="text" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501158 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501162 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501165 2574 flags.go:64] FLAG: --manifest-url="" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501168 2574 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501172 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501175 2574 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501179 2574 flags.go:64] FLAG: --max-pods="110" Nov 02 22:04:50.507048 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501182 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501191 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501194 2574 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501197 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501200 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501203 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501206 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501215 2574 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501218 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501222 2574 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501225 2574 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501228 2574 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501230 2574 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501233 2574 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501236 2574 flags.go:64] FLAG: --port="10250" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501239 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501242 2574 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-051bf643ca3f39370" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501246 2574 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501249 2574 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501252 2574 flags.go:64] FLAG: --register-node="true" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501255 2574 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501258 2574 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501261 2574 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501264 2574 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501267 2574 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501271 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:04:50.507692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501274 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501277 2574 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501280 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501283 2574 flags.go:64] FLAG: --runonce="false" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501286 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501289 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501292 2574 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501295 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501303 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501307 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501310 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501313 2574 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501316 2574 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501319 2574 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501321 2574 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501326 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501329 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501332 2574 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501335 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501340 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501343 2574 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501346 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501353 2574 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501356 2574 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501359 2574 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:04:50.508334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501362 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501365 2574 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501368 2574 flags.go:64] FLAG: --v="2" Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501373 2574 flags.go:64] FLAG: --version="false" Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501377 2574 flags.go:64] FLAG: --vmodule="" Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501381 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501385 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501600 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501604 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501607 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501610 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501613 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501615 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501618 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501620 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501622 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501631 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501634 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501637 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501639 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501642 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.509532 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501644 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501650 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501653 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501656 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501658 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501661 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501663 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501666 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501669 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501671 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501673 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501676 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501679 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501681 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501683 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501686 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501689 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501692 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501694 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.510425 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501696 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501699 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501701 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501704 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501706 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501709 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501712 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501714 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501717 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501725 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501728 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501730 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501733 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501735 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501740 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501743 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501746 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501749 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501752 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501754 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.511229 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501757 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501760 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501762 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501765 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501767 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501769 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501772 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501774 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501777 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501779 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501783 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501786 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501789 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501791 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501794 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501797 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501799 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501802 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501804 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.511774 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501806 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501809 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501811 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501819 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501822 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501824 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501827 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501832 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501834 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501837 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501839 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501842 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501844 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501849 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501852 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501854 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501857 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501859 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.512233 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501863 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501867 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501870 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501872 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501875 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501877 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501880 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501883 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501885 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501888 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501890 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501892 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501895 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501897 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501900 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.501927 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:04:50.512719 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501980 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501990 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501994 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.501996 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502001 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502004 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502007 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502010 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502012 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502015 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502017 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502020 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502022 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502025 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502027 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502030 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502032 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502035 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502037 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502040 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.513111 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502042 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502045 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502047 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502049 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502058 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502061 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502063 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502066 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502068 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502072 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502076 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502079 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502081 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502084 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502092 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502095 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502098 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502101 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502103 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.513606 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502106 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502108 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502110 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502113 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502115 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502118 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502120 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502122 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502125 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502128 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502130 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502133 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502135 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502138 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502140 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502142 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502145 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502147 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502152 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.514072 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502156 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502159 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502162 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502165 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502167 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502170 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502173 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502176 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502178 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502186 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502190 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502193 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502196 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502198 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502201 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502203 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502206 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502208 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502210 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502213 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.514529 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502215 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502217 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502220 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502222 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502225 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502227 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502230 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502232 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502234 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502237 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502239 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502242 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502244 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502247 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502249 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502251 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502254 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502256 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502259 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.515028 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502261 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502263 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502266 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502275 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502278 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502281 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502284 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502286 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.502289 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.511741 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.511757 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.511778 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.511783 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511844 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511851 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511854 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511857 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511861 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511863 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511867 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.515582 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511869 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511872 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511875 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511877 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511880 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511882 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511885 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511888 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511891 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511893 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511896 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511900 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511902 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511905 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511908 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511910 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511913 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511915 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511918 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.516149 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511920 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511923 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511926 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511929 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511931 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511934 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511937 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511940 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511942 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511945 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511947 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511950 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511953 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511955 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511958 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511960 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511963 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511965 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511968 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.516718 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511970 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511974 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511976 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511979 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511982 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511984 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511986 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511989 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511992 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511994 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511996 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.511999 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512002 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512004 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512007 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512010 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512012 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512015 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512018 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512021 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.517185 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512023 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512026 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512029 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512031 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512034 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512036 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512039 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512041 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512044 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512046 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512049 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512052 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512057 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512061 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512065 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512069 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512071 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512074 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.517673 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512077 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512079 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512082 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512084 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512087 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512089 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512092 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512094 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512097 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512100 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512103 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512105 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512108 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512111 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512113 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512116 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512118 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512121 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512123 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512126 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.518101 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512129 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512131 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512134 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.512162 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512310 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512316 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512319 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512323 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512325 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512328 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512331 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512334 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512336 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512339 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512341 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512344 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.518590 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512347 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512349 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512352 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512354 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512357 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512359 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512362 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512364 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512367 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512370 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512372 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512375 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512378 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512380 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512383 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512385 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512388 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512390 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512393 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.518997 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512396 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512398 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512401 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512403 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512406 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512409 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512411 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512414 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512417 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512419 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512422 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512424 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512427 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512429 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512431 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512434 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512436 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512439 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512441 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512445 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.519450 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512447 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512449 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512453 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512456 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512459 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512461 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512463 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512466 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512469 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512471 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512474 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512477 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512479 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512482 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512484 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512486 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512489 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512492 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512494 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.519962 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512497 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512501 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512504 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512507 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512509 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512512 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512514 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512517 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512520 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512522 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512524 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512527 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512530 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512532 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512535 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512537 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512540 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512542 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512545 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512547 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.520407 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512550 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512578 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512583 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512587 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512591 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512594 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512597 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512599 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512602 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512604 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512607 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512610 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512612 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512615 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512618 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.520909 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512621 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.512646 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512681 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512685 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512688 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512691 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512693 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512696 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512698 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512701 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512705 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512707 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512710 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512712 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512715 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:04:50.521272 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512717 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512720 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512723 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512725 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512728 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512731 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512733 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512736 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512739 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512741 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512744 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512747 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512749 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512752 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512755 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512757 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512760 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512763 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512765 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:04:50.521639 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512768 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512771 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512773 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512775 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512778 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512781 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512783 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512785 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512788 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512791 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512793 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512796 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512798 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512801 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512804 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512806 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512809 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512811 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512814 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512816 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512818 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:04:50.522164 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512821 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512823 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512830 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512832 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512835 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512838 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512841 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512843 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512849 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512852 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512855 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512858 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512861 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512863 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512866 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512868 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512871 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512874 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:04:50.522756 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512877 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512879 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512882 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512884 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512887 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512890 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512892 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512895 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512897 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512901 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512905 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512909 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512911 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512914 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512916 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512920 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512923 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512925 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512928 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:04:50.523201 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512930 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512933 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512936 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512938 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512940 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512943 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512946 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512948 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512951 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512953 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512956 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512958 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512963 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512968 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:50.512971 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.513035 2574 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.515342 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.516547 2574 server.go:1014] "Starting client certificate rotation" Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.516700 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:04:50.523713 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.516737 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:04:50.524170 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.518874 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:04:50.524170 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.522101 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:04:50.535485 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.535463 2574 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:04:50.539614 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.539596 2574 log.go:25] "Validated CRI v1 image API" Nov 02 22:04:50.541425 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.541399 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:04:50.548681 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.548596 2574 fs.go:135] Filesystem UUIDs: map[466778fa-b7a7-4ff6-be98-1b6516199dc5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a0445363-335d-4c8f-8c60-63c9a8e95090:/dev/nvme0n1p3] Nov 02 22:04:50.548681 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.548632 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 22:04:50.555592 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.555444 2574 manager.go:217] Machine: {Timestamp:2026-11-02 22:04:50.554114727 +0000 UTC m=+0.382953856 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099073 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254c6bc94ee193eb0c96c09840c9f5 SystemUUID:ec254c6b-c94e-e193-eb0c-96c09840c9f5 BootID:7a064eec-d7cf-4bc1-8456-ee1972e997fc Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e5:24:01:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e5:24:01:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:5b:6a:55:85:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:04:50.555700 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.555598 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:04:50.555766 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.555748 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:04:50.557231 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.557194 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:04:50.557397 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.557232 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-75.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:04:50.557449 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.557400 2574 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:04:50.557449 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.557408 2574 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:04:50.557449 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.557432 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:04:50.557526 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.557448 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:04:50.558371 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.558358 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:04:50.562903 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.562886 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:04:50.563053 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.563042 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:04:50.563822 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.563811 2574 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:04:50.563856 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.563835 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:04:50.563856 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.563847 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:04:50.563938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.563858 2574 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:04:50.563938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.563867 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:04:50.566255 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.566214 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:04:50.570057 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.570029 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:04:50.575491 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.575475 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:04:50.576278 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576265 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:04:50.576384 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576297 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:04:50.576469 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576458 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:04:50.576498 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576475 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:04:50.576498 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576482 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:04:50.576498 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576487 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:04:50.576498 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576493 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:04:50.576498 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.576500 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:04:50.577290 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.577280 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:04:50.577326 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.577291 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:04:50.577326 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.577299 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:04:50.577326 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.577305 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:04:50.577326 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.577315 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:04:50.577326 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.577325 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:04:50.578014 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.577996 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:04:50.578014 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.578011 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:04:50.582078 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.582054 2574 server.go:1263] "Started kubelet" Nov 02 22:04:50.582551 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.582489 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:04:50.582788 ip-10-0-143-75 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:04:50.585129 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.585112 2574 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:04:50.586008 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.585952 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:04:50.586079 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.586033 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:04:50.588250 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.588235 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:04:50.590713 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.590696 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:04:50.590803 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.590743 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:04:50.590854 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.590804 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:50.591380 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.591346 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 22:04:50.591597 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.591577 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:04:50.591692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.591589 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:04:50.591692 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.591694 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:04:50.591834 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.591782 2574 factory.go:55] Registering systemd factory Nov 02 22:04:50.591834 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.591799 2574 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:04:50.591966 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.591952 2574 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:04:50.592010 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.591967 2574 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:04:50.592047 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.592007 2574 factory.go:153] Registering CRI-O factory Nov 02 22:04:50.592047 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.592027 2574 factory.go:223] Registration of the crio container factory successfully Nov 02 22:04:50.592109 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.592085 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:04:50.592109 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.592106 2574 factory.go:103] Registering Raw factory Nov 02 22:04:50.592163 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.592125 2574 manager.go:1199] Started watching for new ooms in manager Nov 02 22:04:50.593141 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.593125 2574 manager.go:319] Starting recovery of all containers Nov 02 22:04:50.603082 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.603059 2574 manager.go:324] Recovery completed Nov 02 22:04:50.608411 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.608395 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:50.610400 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.610380 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:50.610486 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.610415 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:50.610486 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.610430 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:50.611089 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.611076 2574 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:04:50.611089 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.611088 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:04:50.611167 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.611107 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:04:50.611167 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.611079 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:50.611996 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.611979 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:50.612051 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.612012 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:50.612051 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.612028 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:50.613234 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.613223 2574 policy_none.go:50] "Start" Nov 02 22:04:50.613269 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.613239 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:04:50.613269 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.613249 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:04:50.615120 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.615109 2574 policy_none.go:44] "Start" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.660289 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.665252 2574 manager.go:341] "Starting Device Plugin manager" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.665286 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.665300 2574 server.go:87] "Starting device plugin registration server" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.665631 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.665644 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.665752 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.665846 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.665855 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.666418 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.666475 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.668432 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.669800 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.669819 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.669842 2574 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:04:50.678362 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.669881 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 22:04:50.718900 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.718825 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-75.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 22:04:50.719843 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.718837 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-75.us-west-2.compute.internal.18e4599ee5008b85 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-75.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-75.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:04:50.582023045 +0000 UTC m=+0.410862174,LastTimestamp:2026-11-02 22:04:50.582023045 +0000 UTC m=+0.410862174,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-75.us-west-2.compute.internal,}" Nov 02 22:04:50.763776 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.763745 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-75.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 22:04:50.765990 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.765948 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:50.767013 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.766996 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:50.767074 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.767028 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:50.767074 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.767040 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:50.767074 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.767069 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.770276 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.770257 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal"] Nov 02 22:04:50.770326 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.770317 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:50.771194 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.771175 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:50.771273 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.771207 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:50.771273 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.771218 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:50.772962 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.772949 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:50.773099 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.773135 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773119 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:50.773813 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773796 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:50.773813 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773806 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:50.773911 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773827 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:50.773911 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773841 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:50.773911 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773827 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:50.773995 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.773920 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:50.775290 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.775271 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.775382 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.775300 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:50.776024 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.776007 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:50.776103 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.776038 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:50.776103 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.776052 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:50.779706 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.779682 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-75.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 22:04:50.779706 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.779691 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 22:04:50.779863 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.779686 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-75.us-west-2.compute.internal.18e4599ee6b189e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-75.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-75.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-75.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:04:50.610399713 +0000 UTC m=+0.439238845,LastTimestamp:2026-11-02 22:04:50.610399713 +0000 UTC m=+0.439238845,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-75.us-west-2.compute.internal,}" Nov 02 22:04:50.792817 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.792791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.792938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.792821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52559cd016dd3e2df41639aa852bc053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"52559cd016dd3e2df41639aa852bc053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.792938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.792840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.792938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.792853 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:04:50.792938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.792903 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bjdds" Nov 02 22:04:50.807185 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.807156 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.810633 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.810616 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.894176 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.894143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.894176 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.894176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.894373 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.894194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52559cd016dd3e2df41639aa852bc053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"52559cd016dd3e2df41639aa852bc053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.894373 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.894231 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.894373 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.894240 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.894373 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.894275 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52559cd016dd3e2df41639aa852bc053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"52559cd016dd3e2df41639aa852bc053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.938434 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.938405 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:04:50.991447 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.991376 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:04:50.991970 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:50.991953 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:50.992014 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:50.991979 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-75.us-west-2.compute.internal\": node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.111042 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.111007 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:51.112752 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.112728 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.115759 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.115741 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:51.213791 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.213759 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.314615 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.314509 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.415149 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.415106 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.515924 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.515879 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.515924 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.515892 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 22:04:51.516455 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.516062 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:04:51.516455 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.516072 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:04:51.516455 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.516075 2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ac0a2f50c2ab94c12807d3b1c760afba-8cfc36a42c20be03.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-75.us-west-2.compute.internal.18e4599ee6b21169\": read tcp 10.0.143.75:59874->3.131.201.109:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-75.us-west-2.compute.internal.18e4599ee6b21169 default 4998 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-75.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-75.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-75.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:04:50 +0000 UTC,LastTimestamp:2026-11-02 22:04:50.767044231 +0000 UTC m=+0.595883360,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-75.us-west-2.compute.internal,}" Nov 02 22:04:51.616957 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.616888 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.717641 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.717606 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.797385 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.797337 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:59:50 +0000 UTC" deadline="2028-07-05 20:56:47.776368372 +0000 UTC" Nov 02 22:04:51.797385 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.797373 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14662h51m55.978998808s" Nov 02 22:04:51.801605 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.801588 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:04:51.802644 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.802624 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:04:51.802753 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.802659 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:04:51.802753 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.802677 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:04:51.818235 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.818201 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.887633 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.887530 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:04:51.918824 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:51.918788 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:04:51.938274 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.938250 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:04:51.992081 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:51.992053 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:52.057161 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.057125 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:04:52.058219 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.058203 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:04:52.123145 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.123121 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:04:52.284482 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:52.284441 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c23d8b6ee5b61bb2ffa790b47abb66.slice/crio-6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7 WatchSource:0}: Error finding container 6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7: Status 404 returned error can't find the container with id 6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7 Nov 02 22:04:52.284948 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:52.284933 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52559cd016dd3e2df41639aa852bc053.slice/crio-7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773 WatchSource:0}: Error finding container 7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773: Status 404 returned error can't find the container with id 7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773 Nov 02 22:04:52.291169 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.291150 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:04:52.567369 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.567285 2574 apiserver.go:52] "Watching apiserver" Nov 02 22:04:52.575185 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.575159 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:04:52.592323 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.592290 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 22:04:52.675511 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.675456 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"87c23d8b6ee5b61bb2ffa790b47abb66","Type":"ContainerStarted","Data":"6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7"} Nov 02 22:04:52.676608 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.676585 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"52559cd016dd3e2df41639aa852bc053","Type":"ContainerStarted","Data":"7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773"} Nov 02 22:04:52.736050 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.736020 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:04:52.736359 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.736341 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-55xk7","openshift-ovn-kubernetes/ovnkube-node-5xqr8","openshift-multus/multus-xf2bk","openshift-multus/network-metrics-daemon-h7lsc","openshift-network-operator/iptables-alerter-47zpv","kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal","openshift-dns/node-resolver-6lwq6","openshift-image-registry/node-ca-lsbsf","openshift-network-diagnostics/network-check-target-pb5sz","kube-system/global-pull-secret-syncer-knp2v","kube-system/konnectivity-agent-zzcqw","openshift-cluster-node-tuning-operator/tuned-n9jzr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal"] Nov 02 22:04:52.740391 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.740373 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:52.740498 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.740454 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:04:52.742683 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.742665 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.744990 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.744960 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.747493 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.747472 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.750573 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.750554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.752959 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.752943 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.757368 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.757346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.759894 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.759873 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:52.759993 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.759960 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:04:52.762199 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.762182 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.764410 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.764390 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:04:52.764599 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.764563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.764672 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.764653 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:04:52.766934 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.766917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:52.769340 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.769322 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.792724 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.792703 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:04:52.800370 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.800340 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:04:52.800370 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.800347 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:04:52.800521 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.800347 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:04:52.800521 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.800379 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-klqgw\"" Nov 02 22:04:52.800521 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.800421 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:04:52.800521 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.800436 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:04:52.800766 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.800753 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:04:52.805386 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.805447 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.805447 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvddc\" (UniqueName: \"kubernetes.io/projected/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kube-api-access-vvddc\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.805447 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:52.805549 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805483 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-netns\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.805549 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.805549 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jph7w\" (UniqueName: \"kubernetes.io/projected/ed940156-2f94-4372-a949-f07e265f0c8b-kube-api-access-jph7w\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.805668 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-var-lib-kubelet\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.805668 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-host\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.805668 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-systemd-units\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.805761 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-etc-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.805761 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysconfig\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.805761 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-node-log\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.805894 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovn-node-metrics-cert\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.805894 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:52.805894 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzt5g\" (UniqueName: \"kubernetes.io/projected/306b1470-ddb8-4e0c-b450-3c2ddb541faf-kube-api-access-rzt5g\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.805894 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-ovn\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.805894 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-netd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.806038 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.806038 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-kubelet-config\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.806038 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-modprobe-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.806038 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.806038 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.805982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-netns\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806038 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806016 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-multus\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-daemon-config\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-etc-kubernetes\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-tmp\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-socket-dir-parent\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-k8s-cni-cncf-io\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-cnibin\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.806208 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-agent-certs\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-bin\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-conf-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-host\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806292 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-device-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-system-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-kubelet\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-kubernetes\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-systemd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.806489 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-script-lib\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-system-cni-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-tuned\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-bin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-hostroot\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-slash\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cni-binary-copy\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-run\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgw2f\" (UniqueName: \"kubernetes.io/projected/22f84c42-8ee4-4133-a636-f02598d2b61b-kube-api-access-vgw2f\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfv6\" (UniqueName: \"kubernetes.io/projected/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-kube-api-access-hcfv6\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-env-overrides\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk29s\" (UniqueName: \"kubernetes.io/projected/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-kube-api-access-dk29s\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.807034 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-lib-modules\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.806964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-host-slash\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsrhp\" (UniqueName: \"kubernetes.io/projected/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-kube-api-access-wsrhp\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-iptables-alerter-script\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-kubelet\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-config\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-conf\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f84c42-8ee4-4133-a636-f02598d2b61b-tmp-dir\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-var-lib-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktm5\" (UniqueName: \"kubernetes.io/projected/d6ed1d10-fcee-4a00-a544-efb9346dac66-kube-api-access-8ktm5\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cnibin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-multus-certs\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-systemd\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-sys\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f84c42-8ee4-4133-a636-f02598d2b61b-hosts-file\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.807646 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807304 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-serviceca\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.808126 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-log-socket\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.808126 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwps\" (UniqueName: \"kubernetes.io/projected/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-kube-api-access-hlwps\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.808126 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-dbus\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.808126 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.807466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-konnectivity-ca\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:52.818223 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.818174 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:04:52.818223 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.818205 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:04:52.818326 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.818282 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrhfb\"" Nov 02 22:04:52.819141 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.819096 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.819648 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.819662 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.819790 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.819806 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.819894 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcntw\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.819934 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.820010 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.820029 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:04:52.820202 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.820079 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rksbh\"" Nov 02 22:04:52.820731 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.820232 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fcg7g\"" Nov 02 22:04:52.820731 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.820243 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:04:52.852966 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.852933 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:04:52.855240 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855208 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4ts5p\"" Nov 02 22:04:52.855240 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855223 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:04:52.855418 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855254 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:04:52.855556 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855534 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:04:52.855691 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855562 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:04:52.855691 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855613 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:04:52.855691 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855654 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:04:52.855691 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855685 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:04:52.855867 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855772 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-656w2\"" Nov 02 22:04:52.855867 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855801 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:04:52.855932 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855878 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4rxp8\"" Nov 02 22:04:52.855932 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.855908 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:04:52.873127 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.873096 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rsfpb\"" Nov 02 22:04:52.874186 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.874167 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:04:52.874284 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.874209 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m2f9j" Nov 02 22:04:52.908375 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrhp\" (UniqueName: \"kubernetes.io/projected/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-kube-api-access-wsrhp\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:52.908505 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.908505 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-iptables-alerter-script\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.908505 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-kubelet\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.908505 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-config\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908525 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908560 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-kubelet\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-conf\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f84c42-8ee4-4133-a636-f02598d2b61b-tmp-dir\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-var-lib-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktm5\" (UniqueName: \"kubernetes.io/projected/d6ed1d10-fcee-4a00-a544-efb9346dac66-kube-api-access-8ktm5\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.908706 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cnibin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908718 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-conf\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-multus-certs\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908753 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-systemd\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908757 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-var-lib-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-sys\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908818 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-multus-certs\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908827 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cnibin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f84c42-8ee4-4133-a636-f02598d2b61b-hosts-file\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908863 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-sys\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-serviceca\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908885 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-systemd\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908903 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-log-socket\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwps\" (UniqueName: \"kubernetes.io/projected/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-kube-api-access-hlwps\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908941 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f84c42-8ee4-4133-a636-f02598d2b61b-hosts-file\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-dbus\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908966 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-log-socket\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.909101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.908986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-konnectivity-ca\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvddc\" (UniqueName: \"kubernetes.io/projected/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kube-api-access-vvddc\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909101 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-netns\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909128 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-dbus\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909167 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-netns\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909167 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909174 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jph7w\" (UniqueName: \"kubernetes.io/projected/ed940156-2f94-4372-a949-f07e265f0c8b-kube-api-access-jph7w\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-var-lib-kubelet\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-host\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-systemd-units\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909286 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-var-lib-kubelet\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-etc-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.909981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysconfig\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-node-log\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovn-node-metrics-cert\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909393 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-systemd-units\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzt5g\" (UniqueName: \"kubernetes.io/projected/306b1470-ddb8-4e0c-b450-3c2ddb541faf-kube-api-access-rzt5g\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909453 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-host\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909457 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-etc-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-ovn\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-netd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909510 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-ovn\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-kubelet-config\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.909570 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.909680 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs podName:6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:04:53.409658982 +0000 UTC m=+3.238498116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs") pod "network-metrics-daemon-h7lsc" (UID: "6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-modprobe-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909694 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-modprobe-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.910957 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909749 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysconfig\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909758 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-kubelet-config\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909804 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-netd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909810 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-netns\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909854 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-node-log\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-netns\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909873 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-iptables-alerter-script\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909893 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-multus\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909838 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-config\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909909 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-multus\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-daemon-config\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.909987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-etc-kubernetes\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-tmp\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-socket-dir-parent\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.911863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-k8s-cni-cncf-io\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-cnibin\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-agent-certs\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-bin\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910161 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f84c42-8ee4-4133-a636-f02598d2b61b-tmp-dir\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-conf-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-host\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-device-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-system-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910289 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-serviceca\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910347 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-bin\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-kubelet\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-kubernetes\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-systemd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.912440 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910466 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-script-lib\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910468 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-socket-dir-parent\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-system-cni-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910514 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-conf-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-tuned\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910587 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-daemon-config\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-system-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910654 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-etc-kubernetes\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910589 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-k8s-cni-cncf-io\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910673 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-bin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910587 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-device-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910676 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-cnibin\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910691 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910712 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-kubelet\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-hostroot\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910745 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-hostroot\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910762 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-system-cni-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910719 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-bin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910774 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-systemd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-slash\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cni-binary-copy\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-run\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgw2f\" (UniqueName: \"kubernetes.io/projected/22f84c42-8ee4-4133-a636-f02598d2b61b-kube-api-access-vgw2f\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfv6\" (UniqueName: \"kubernetes.io/projected/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-kube-api-access-hcfv6\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-env-overrides\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910845 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-kubernetes\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk29s\" (UniqueName: \"kubernetes.io/projected/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-kube-api-access-dk29s\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-lib-modules\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.913649 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911065 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-host-slash\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911143 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-host-slash\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.910981 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911190 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911507 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-lib-modules\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911562 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-slash\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911574 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-script-lib\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911609 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-host\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911666 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-run\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.911773 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.911869 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret podName:74da9552-2470-4a9f-8160-d3a29aa5d9f5 nodeName:}" failed. No retries permitted until 2026-11-02 22:04:53.41184975 +0000 UTC m=+3.240688888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret") pod "global-pull-secret-syncer-knp2v" (UID: "74da9552-2470-4a9f-8160-d3a29aa5d9f5") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.911980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cni-binary-copy\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.912045 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-env-overrides\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.913053 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-konnectivity-ca\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.913724 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-tmp\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.914144 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.913796 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-tuned\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.914615 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.913914 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovn-node-metrics-cert\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.914615 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.914269 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-agent-certs\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:52.973663 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.973622 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:04:52.973663 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.973661 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:04:52.973663 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.973676 2574 projected.go:196] Error preparing data for projected volume kube-api-access-kwd52 for pod openshift-network-diagnostics/network-check-target-pb5sz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:52.974025 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:52.973760 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52 podName:e453e98d-c621-4498-ad81-8cbc7eee10fe nodeName:}" failed. No retries permitted until 2026-11-02 22:04:53.473735517 +0000 UTC m=+3.302574659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kwd52" (UniqueName: "kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52") pod "network-check-target-pb5sz" (UID: "e453e98d-c621-4498-ad81-8cbc7eee10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:52.976783 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.976751 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktm5\" (UniqueName: \"kubernetes.io/projected/d6ed1d10-fcee-4a00-a544-efb9346dac66-kube-api-access-8ktm5\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:52.977103 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.976754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwps\" (UniqueName: \"kubernetes.io/projected/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-kube-api-access-hlwps\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:04:52.977461 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.977437 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvddc\" (UniqueName: \"kubernetes.io/projected/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kube-api-access-vvddc\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:52.977570 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.977464 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrhp\" (UniqueName: \"kubernetes.io/projected/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-kube-api-access-wsrhp\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:52.977570 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.977439 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzt5g\" (UniqueName: \"kubernetes.io/projected/306b1470-ddb8-4e0c-b450-3c2ddb541faf-kube-api-access-rzt5g\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:52.977570 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.977510 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jph7w\" (UniqueName: \"kubernetes.io/projected/ed940156-2f94-4372-a949-f07e265f0c8b-kube-api-access-jph7w\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:52.977762 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.977744 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk29s\" (UniqueName: \"kubernetes.io/projected/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-kube-api-access-dk29s\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:52.977844 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.977822 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgw2f\" (UniqueName: \"kubernetes.io/projected/22f84c42-8ee4-4133-a636-f02598d2b61b-kube-api-access-vgw2f\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:52.978548 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:52.978529 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfv6\" (UniqueName: \"kubernetes.io/projected/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-kube-api-access-hcfv6\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:53.068795 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.068711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xf2bk" Nov 02 22:04:53.078939 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.078910 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb314a49f_8d9b_41f7_81f6_67f6c486f9b4.slice/crio-b5303ac1ad52ac0bfe694e1759e2af566d7b4d5bb4d877ece111dd9c71b9ee98 WatchSource:0}: Error finding container b5303ac1ad52ac0bfe694e1759e2af566d7b4d5bb4d877ece111dd9c71b9ee98: Status 404 returned error can't find the container with id b5303ac1ad52ac0bfe694e1759e2af566d7b4d5bb4d877ece111dd9c71b9ee98 Nov 02 22:04:53.079047 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.078946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:04:53.082720 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.082703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:04:53.086334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.086313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:04:53.087454 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.087429 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d87cc6_0ee0_44bc_9a3e_43ea8fefbfed.slice/crio-89e7a1e5f423c963456d66b6293e7c2c5220df7af479ca412e548dc1205b2fcd WatchSource:0}: Error finding container 89e7a1e5f423c963456d66b6293e7c2c5220df7af479ca412e548dc1205b2fcd: Status 404 returned error can't find the container with id 89e7a1e5f423c963456d66b6293e7c2c5220df7af479ca412e548dc1205b2fcd Nov 02 22:04:53.088005 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.087987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:04:53.092835 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.092813 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ed1d10_fcee_4a00_a544_efb9346dac66.slice/crio-cd19933d1a573d9cdaec74f596f7fa7384455d80c4b43f431c28ef0bee62ac0e WatchSource:0}: Error finding container cd19933d1a573d9cdaec74f596f7fa7384455d80c4b43f431c28ef0bee62ac0e: Status 404 returned error can't find the container with id cd19933d1a573d9cdaec74f596f7fa7384455d80c4b43f431c28ef0bee62ac0e Nov 02 22:04:53.096200 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.096185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:04:53.097643 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.097623 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306b1470_ddb8_4e0c_b450_3c2ddb541faf.slice/crio-a008e71cfa7b32193a6eb0c3489a0b58201f674cad6550c3ba13be928b46a521 WatchSource:0}: Error finding container a008e71cfa7b32193a6eb0c3489a0b58201f674cad6550c3ba13be928b46a521: Status 404 returned error can't find the container with id a008e71cfa7b32193a6eb0c3489a0b58201f674cad6550c3ba13be928b46a521 Nov 02 22:04:53.098577 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.098532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:04:53.099418 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.099398 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c42_8ee4_4133_a636_f02598d2b61b.slice/crio-f481cf99c651eee61f335e49d463ec1d951374b2c5058f77e8c0c48cbe8a661f WatchSource:0}: Error finding container f481cf99c651eee61f335e49d463ec1d951374b2c5058f77e8c0c48cbe8a661f: Status 404 returned error can't find the container with id f481cf99c651eee61f335e49d463ec1d951374b2c5058f77e8c0c48cbe8a661f Nov 02 22:04:53.101056 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.101038 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:53.106761 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.106739 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded940156_2f94_4372_a949_f07e265f0c8b.slice/crio-4e2290f26a1c7b0a1235abbf0965ef1f5bb30cf0a16f024547401fb76a29890b WatchSource:0}: Error finding container 4e2290f26a1c7b0a1235abbf0965ef1f5bb30cf0a16f024547401fb76a29890b: Status 404 returned error can't find the container with id 4e2290f26a1c7b0a1235abbf0965ef1f5bb30cf0a16f024547401fb76a29890b Nov 02 22:04:53.107662 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.107641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:04:53.109128 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.109110 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3ac303_6d4c_4f0a_9279_2b3c428adccd.slice/crio-60d62e41334654822970b4cda017d2643cf56db8869f668311d8826c21bbabd6 WatchSource:0}: Error finding container 60d62e41334654822970b4cda017d2643cf56db8869f668311d8826c21bbabd6: Status 404 returned error can't find the container with id 60d62e41334654822970b4cda017d2643cf56db8869f668311d8826c21bbabd6 Nov 02 22:04:53.112264 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.112245 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5806a44_0c3e_4ea8_a72c_a8fe4a2aea7a.slice/crio-051c1af1375591ce8b9003110ee894ae74962a0d86f33ded706a979592c68342 WatchSource:0}: Error finding container 051c1af1375591ce8b9003110ee894ae74962a0d86f33ded706a979592c68342: Status 404 returned error can't find the container with id 051c1af1375591ce8b9003110ee894ae74962a0d86f33ded706a979592c68342 Nov 02 22:04:53.117000 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:04:53.116980 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7648ae20_ac24_49f7_a8a7_11b204b9a2b3.slice/crio-e169284b4a8e158327a8653f56ac3081327a00ad08954dab565ecc839277bed5 WatchSource:0}: Error finding container e169284b4a8e158327a8653f56ac3081327a00ad08954dab565ecc839277bed5: Status 404 returned error can't find the container with id e169284b4a8e158327a8653f56ac3081327a00ad08954dab565ecc839277bed5 Nov 02 22:04:53.414479 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.414389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:53.414479 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.414457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:53.414730 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.414555 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:53.414730 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.414635 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:53.414730 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.414642 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs podName:6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:04:54.414625142 +0000 UTC m=+4.243464293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs") pod "network-metrics-daemon-h7lsc" (UID: "6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:53.414730 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.414703 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret podName:74da9552-2470-4a9f-8160-d3a29aa5d9f5 nodeName:}" failed. No retries permitted until 2026-11-02 22:04:54.414686092 +0000 UTC m=+4.243525215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret") pod "global-pull-secret-syncer-knp2v" (UID: "74da9552-2470-4a9f-8160-d3a29aa5d9f5") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:53.515796 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.515754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:53.515978 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.515948 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:04:53.515978 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.515983 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:04:53.516149 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.515994 2574 projected.go:196] Error preparing data for projected volume kube-api-access-kwd52 for pod openshift-network-diagnostics/network-check-target-pb5sz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:53.516149 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:53.516050 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52 podName:e453e98d-c621-4498-ad81-8cbc7eee10fe nodeName:}" failed. No retries permitted until 2026-11-02 22:04:54.516030271 +0000 UTC m=+4.344869401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwd52" (UniqueName: "kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52") pod "network-check-target-pb5sz" (UID: "e453e98d-c621-4498-ad81-8cbc7eee10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:53.680423 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.680336 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" event={"ID":"7648ae20-ac24-49f7-a8a7-11b204b9a2b3","Type":"ContainerStarted","Data":"e169284b4a8e158327a8653f56ac3081327a00ad08954dab565ecc839277bed5"} Nov 02 22:04:53.681844 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.681811 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lsbsf" event={"ID":"cc3ac303-6d4c-4f0a-9279-2b3c428adccd","Type":"ContainerStarted","Data":"60d62e41334654822970b4cda017d2643cf56db8869f668311d8826c21bbabd6"} Nov 02 22:04:53.683212 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.683176 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerStarted","Data":"4e2290f26a1c7b0a1235abbf0965ef1f5bb30cf0a16f024547401fb76a29890b"} Nov 02 22:04:53.684740 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.684713 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6lwq6" event={"ID":"22f84c42-8ee4-4133-a636-f02598d2b61b","Type":"ContainerStarted","Data":"f481cf99c651eee61f335e49d463ec1d951374b2c5058f77e8c0c48cbe8a661f"} Nov 02 22:04:53.686010 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.685986 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-47zpv" event={"ID":"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed","Type":"ContainerStarted","Data":"89e7a1e5f423c963456d66b6293e7c2c5220df7af479ca412e548dc1205b2fcd"} Nov 02 22:04:53.687318 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.687290 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zzcqw" event={"ID":"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a","Type":"ContainerStarted","Data":"051c1af1375591ce8b9003110ee894ae74962a0d86f33ded706a979592c68342"} Nov 02 22:04:53.688671 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.688648 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" event={"ID":"306b1470-ddb8-4e0c-b450-3c2ddb541faf","Type":"ContainerStarted","Data":"a008e71cfa7b32193a6eb0c3489a0b58201f674cad6550c3ba13be928b46a521"} Nov 02 22:04:53.690159 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.690136 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"cd19933d1a573d9cdaec74f596f7fa7384455d80c4b43f431c28ef0bee62ac0e"} Nov 02 22:04:53.691511 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.691490 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xf2bk" event={"ID":"b314a49f-8d9b-41f7-81f6-67f6c486f9b4","Type":"ContainerStarted","Data":"b5303ac1ad52ac0bfe694e1759e2af566d7b4d5bb4d877ece111dd9c71b9ee98"} Nov 02 22:04:53.877849 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.877783 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:59:52 +0000 UTC" deadline="2028-07-20 13:55:18.350441782 +0000 UTC" Nov 02 22:04:53.877849 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:53.877844 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15015h50m24.472601896s" Nov 02 22:04:54.423904 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.423822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:54.423904 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.423882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:54.424133 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.424040 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:54.424133 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.424105 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret podName:74da9552-2470-4a9f-8160-d3a29aa5d9f5 nodeName:}" failed. No retries permitted until 2026-11-02 22:04:56.424086104 +0000 UTC m=+6.252925236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret") pod "global-pull-secret-syncer-knp2v" (UID: "74da9552-2470-4a9f-8160-d3a29aa5d9f5") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:54.424256 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.424180 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:54.424256 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.424230 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs podName:6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:04:56.424217013 +0000 UTC m=+6.253056142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs") pod "network-metrics-daemon-h7lsc" (UID: "6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:54.524841 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.524800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:54.525046 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.525018 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:04:54.525046 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.525042 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:04:54.525141 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.525055 2574 projected.go:196] Error preparing data for projected volume kube-api-access-kwd52 for pod openshift-network-diagnostics/network-check-target-pb5sz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:54.525141 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.525123 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52 podName:e453e98d-c621-4498-ad81-8cbc7eee10fe nodeName:}" failed. No retries permitted until 2026-11-02 22:04:56.525103126 +0000 UTC m=+6.353942244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwd52" (UniqueName: "kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52") pod "network-check-target-pb5sz" (UID: "e453e98d-c621-4498-ad81-8cbc7eee10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:54.673448 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.673414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:54.673639 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.673558 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:04:54.673758 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.673679 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:54.673758 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.673748 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:04:54.673859 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.673819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:54.674003 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:54.673916 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:04:54.878246 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.878213 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:59:52 +0000 UTC" deadline="2028-03-31 15:52:15.681917495 +0000 UTC" Nov 02 22:04:54.878246 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:54.878245 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12353h47m20.803676211s" Nov 02 22:04:56.443125 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.442866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:56.443596 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.443199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:56.443596 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.443340 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:56.443596 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.443402 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs podName:6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:00.443382673 +0000 UTC m=+10.272221794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs") pod "network-metrics-daemon-h7lsc" (UID: "6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:04:56.443596 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.443463 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:56.443596 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.443495 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret podName:74da9552-2470-4a9f-8160-d3a29aa5d9f5 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:00.44348386 +0000 UTC m=+10.272322991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret") pod "global-pull-secret-syncer-knp2v" (UID: "74da9552-2470-4a9f-8160-d3a29aa5d9f5") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:04:56.545254 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.544934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:56.545254 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.545115 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:04:56.545254 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.545139 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:04:56.545254 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.545152 2574 projected.go:196] Error preparing data for projected volume kube-api-access-kwd52 for pod openshift-network-diagnostics/network-check-target-pb5sz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:56.545254 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.545212 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52 podName:e453e98d-c621-4498-ad81-8cbc7eee10fe nodeName:}" failed. No retries permitted until 2026-11-02 22:05:00.545193425 +0000 UTC m=+10.374032546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwd52" (UniqueName: "kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52") pod "network-check-target-pb5sz" (UID: "e453e98d-c621-4498-ad81-8cbc7eee10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:04:56.670779 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.670698 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:56.670956 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.670848 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:04:56.671025 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.670963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:56.671332 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.671072 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:04:56.671332 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.671182 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:56.671332 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:56.671285 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:04:56.714913 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.714864 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"52559cd016dd3e2df41639aa852bc053","Type":"ContainerStarted","Data":"9a2bad957f7798870503b821c438f36b8b74e1502ad4d2245f134afd6e235205"} Nov 02 22:04:56.725734 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.725695 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="afdf8f5dd2874da460d1e31cb7e7ed1dc698701392de78ba5914f96a42e4d5fd" exitCode=0 Nov 02 22:04:56.725897 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.725802 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerDied","Data":"afdf8f5dd2874da460d1e31cb7e7ed1dc698701392de78ba5914f96a42e4d5fd"} Nov 02 22:04:56.730630 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.729979 2574 generic.go:356] "Generic (PLEG): container finished" podID="87c23d8b6ee5b61bb2ffa790b47abb66" containerID="fc93a35565f4a587049cca83ef1faa9cb191a7b3564712f26ec479f1fd0a2beb" exitCode=0 Nov 02 22:04:56.730630 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.730027 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"87c23d8b6ee5b61bb2ffa790b47abb66","Type":"ContainerDied","Data":"fc93a35565f4a587049cca83ef1faa9cb191a7b3564712f26ec479f1fd0a2beb"} Nov 02 22:04:56.833730 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:56.833285 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" podStartSLOduration=4.833264027 podStartE2EDuration="4.833264027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:56.832261737 +0000 UTC m=+6.661100873" watchObservedRunningTime="2026-11-02 22:04:56.833264027 +0000 UTC m=+6.662103165" Nov 02 22:04:57.741408 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:57.741371 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"87c23d8b6ee5b61bb2ffa790b47abb66","Type":"ContainerStarted","Data":"84a78d3391b35b5c1ba1abf01bebcffa3eb51c89fd756e9c08470cfdac0d7386"} Nov 02 22:04:58.672870 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:58.672839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:04:58.673078 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:58.673001 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:04:58.673135 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:58.673080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:04:58.673186 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:58.673152 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:04:58.673224 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:58.673216 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:04:58.673309 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:04:58.673283 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:04:58.746709 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:58.746631 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zzcqw" event={"ID":"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a","Type":"ContainerStarted","Data":"d899d683c56c0e717b43365b24b2c60454de3e0b9809226fae06e0ad1eb1f351"} Nov 02 22:04:58.866170 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:58.865070 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zzcqw" podStartSLOduration=3.318301337 podStartE2EDuration="7.865053049s" totalImagesPullingTime="4.546751712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:58.864998413 +0000 UTC m=+8.693837552" watchObservedRunningTime="2026-11-02 22:04:58.865053049 +0000 UTC m=+8.693892188" Nov 02 22:04:58.866170 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:58.865382 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" podStartSLOduration=6.86537111 podStartE2EDuration="6.86537111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:57.861483754 +0000 UTC m=+7.690322894" watchObservedRunningTime="2026-11-02 22:04:58.86537111 +0000 UTC m=+8.694210251" Nov 02 22:04:59.554017 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:59.553970 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:59.554951 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:59.554902 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:59.753695 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:59.753659 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lsbsf" event={"ID":"cc3ac303-6d4c-4f0a-9279-2b3c428adccd","Type":"ContainerStarted","Data":"a5840140af2fc2ec86aee3693f68f15b9bfd6137c12395dc06d6ede37d32a1d9"} Nov 02 22:04:59.865268 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:59.865246 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:59.865896 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:59.865879 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:04:59.992055 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:04:59.991978 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lsbsf" podStartSLOduration=3.164387283 podStartE2EDuration="8.991955624s" totalImagesPullingTime="5.827568341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:59.873065579 +0000 UTC m=+9.701904718" watchObservedRunningTime="2026-11-02 22:04:59.991955624 +0000 UTC m=+9.820794763" Nov 02 22:05:00.477863 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:00.477811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:00.478035 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:00.477915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:00.478035 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.477951 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:00.478035 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.478028 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:00.478189 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.478034 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs podName:6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:08.478012675 +0000 UTC m=+18.306851801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs") pod "network-metrics-daemon-h7lsc" (UID: "6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:00.478189 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.478083 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret podName:74da9552-2470-4a9f-8160-d3a29aa5d9f5 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:08.478071836 +0000 UTC m=+18.306910953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret") pod "global-pull-secret-syncer-knp2v" (UID: "74da9552-2470-4a9f-8160-d3a29aa5d9f5") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:00.579307 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:00.579267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:00.579556 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.579532 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:00.579648 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.579586 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:00.579648 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.579601 2574 projected.go:196] Error preparing data for projected volume kube-api-access-kwd52 for pod openshift-network-diagnostics/network-check-target-pb5sz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:00.579761 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.579669 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52 podName:e453e98d-c621-4498-ad81-8cbc7eee10fe nodeName:}" failed. No retries permitted until 2026-11-02 22:05:08.579647859 +0000 UTC m=+18.408486991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwd52" (UniqueName: "kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52") pod "network-check-target-pb5sz" (UID: "e453e98d-c621-4498-ad81-8cbc7eee10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:00.673806 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:00.672545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:00.673806 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.672696 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:00.673806 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:00.672899 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:00.673806 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.672988 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:00.673806 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:00.673035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:00.673806 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:00.673101 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:00.766067 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:00.765480 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" event={"ID":"7648ae20-ac24-49f7-a8a7-11b204b9a2b3","Type":"ContainerStarted","Data":"aa5c9a40498824e9f9467b52c30e58b3e40352fc4c08847a5995da8d0aecd573"} Nov 02 22:05:02.105374 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.105346 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:05:02.670702 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.670661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:02.670892 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.670727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:02.670892 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.670725 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:02.670997 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:02.670912 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:02.671050 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:02.670992 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:02.671050 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:02.671030 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:02.675245 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.675150 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:05:02.105363306Z","UUID":"796fbcac-ab40-45c7-b907-b3682d3962b8","Handler":null,"Name":"","Endpoint":""} Nov 02 22:05:02.676904 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.676884 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:05:02.677005 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.676912 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:05:02.772299 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.772037 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xf2bk" event={"ID":"b314a49f-8d9b-41f7-81f6-67f6c486f9b4","Type":"ContainerStarted","Data":"11d54d5429bf55ed68f2316c06920edc5060134062015ee49a771bfc17a91924"} Nov 02 22:05:02.775079 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.775031 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" event={"ID":"7648ae20-ac24-49f7-a8a7-11b204b9a2b3","Type":"ContainerStarted","Data":"117e08f89811e8255771c88e2048d387de83dd08078cceeb1c62c9204fefdad1"} Nov 02 22:05:02.777379 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.777340 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="52c91ca65299b795dd70711236d23d755ad7844c27986db7d7663ef8b1c70343" exitCode=0 Nov 02 22:05:02.777514 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.777425 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerDied","Data":"52c91ca65299b795dd70711236d23d755ad7844c27986db7d7663ef8b1c70343"} Nov 02 22:05:02.779849 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.779825 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6lwq6" event={"ID":"22f84c42-8ee4-4133-a636-f02598d2b61b","Type":"ContainerStarted","Data":"8ac9349883a7486b72b20bc64b9da51538542577fbf92efca33a81d871b7a823"} Nov 02 22:05:02.899016 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:02.898949 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xf2bk" podStartSLOduration=4.321300292 podStartE2EDuration="12.898935143s" totalImagesPullingTime="8.577634851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:02.898285508 +0000 UTC m=+12.727124647" watchObservedRunningTime="2026-11-02 22:05:02.898935143 +0000 UTC m=+12.727774280" Nov 02 22:05:03.141485 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:03.141427 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6lwq6" podStartSLOduration=4.522950483 podStartE2EDuration="13.141409012s" totalImagesPullingTime="8.618458529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:03.141343134 +0000 UTC m=+12.970182272" watchObservedRunningTime="2026-11-02 22:05:03.141409012 +0000 UTC m=+12.970248150" Nov 02 22:05:03.784450 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:03.784404 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-47zpv" event={"ID":"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed","Type":"ContainerStarted","Data":"8004a07d978b9744b34ca5d27c24110220badf60c1fc06699660004aa3bd2dab"} Nov 02 22:05:03.903587 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:03.903519 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-47zpv" podStartSLOduration=5.024751366 podStartE2EDuration="13.903505964s" totalImagesPullingTime="8.878754598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:03.90340293 +0000 UTC m=+13.732242070" watchObservedRunningTime="2026-11-02 22:05:03.903505964 +0000 UTC m=+13.732345097" Nov 02 22:05:04.670604 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:04.670553 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:04.670604 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:04.670607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:04.671334 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:04.670556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:04.671334 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:04.670766 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:04.671334 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:04.670818 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:04.671334 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:04.670866 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:04.788627 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:04.788592 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="9b20aaeee09c1817bbf1c994439e58b7c4ae1db1adb24ecde1cc649b12f08cb2" exitCode=0 Nov 02 22:05:04.788805 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:04.788674 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerDied","Data":"9b20aaeee09c1817bbf1c994439e58b7c4ae1db1adb24ecde1cc649b12f08cb2"} Nov 02 22:05:04.791152 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:04.791126 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" event={"ID":"306b1470-ddb8-4e0c-b450-3c2ddb541faf","Type":"ContainerStarted","Data":"4460a59b51f092b53dafb74f2ecb92280cefddf4c206b16cabbf3088f865967c"} Nov 02 22:05:04.794971 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:04.794946 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" event={"ID":"7648ae20-ac24-49f7-a8a7-11b204b9a2b3","Type":"ContainerStarted","Data":"f90852ad4f7f1eef10b663358dca50b1825774466a2f5235d82b84a8be2c74a2"} Nov 02 22:05:05.037494 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:05.037429 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" podStartSLOduration=4.402199736 podStartE2EDuration="15.037413493s" totalImagesPullingTime="10.635213757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:05.036698465 +0000 UTC m=+14.865537606" watchObservedRunningTime="2026-11-02 22:05:05.037413493 +0000 UTC m=+14.866252631" Nov 02 22:05:05.160870 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:05.160817 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" podStartSLOduration=3.835963079 podStartE2EDuration="14.160801369s" totalImagesPullingTime="10.32483829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:05.160459305 +0000 UTC m=+14.989298445" watchObservedRunningTime="2026-11-02 22:05:05.160801369 +0000 UTC m=+14.989640507" Nov 02 22:05:06.670978 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:06.670939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:06.671558 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:06.671081 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:06.671558 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:06.670957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:06.671558 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:06.671191 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:06.671723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:06.671597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:06.671723 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:06.671692 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:06.802167 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:06.802128 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="892efecf0729e302bf14be10c1ad95462ca2d2449f538af519343a3d9fbea367" exitCode=0 Nov 02 22:05:06.802311 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:06.802192 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerDied","Data":"892efecf0729e302bf14be10c1ad95462ca2d2449f538af519343a3d9fbea367"} Nov 02 22:05:08.546955 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.546495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:08.547395 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.547021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:08.547395 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.546754 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:08.547395 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.547140 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs podName:6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:24.547118938 +0000 UTC m=+34.375958055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs") pod "network-metrics-daemon-h7lsc" (UID: "6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:08.547395 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.547146 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:08.547395 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.547197 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret podName:74da9552-2470-4a9f-8160-d3a29aa5d9f5 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:24.547181698 +0000 UTC m=+34.376020832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret") pod "global-pull-secret-syncer-knp2v" (UID: "74da9552-2470-4a9f-8160-d3a29aa5d9f5") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:08.648366 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.648335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:08.648525 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.648487 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:08.648525 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.648505 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:08.648525 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.648519 2574 projected.go:196] Error preparing data for projected volume kube-api-access-kwd52 for pod openshift-network-diagnostics/network-check-target-pb5sz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:08.648655 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.648593 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52 podName:e453e98d-c621-4498-ad81-8cbc7eee10fe nodeName:}" failed. No retries permitted until 2026-11-02 22:05:24.648559655 +0000 UTC m=+34.477398784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwd52" (UniqueName: "kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52") pod "network-check-target-pb5sz" (UID: "e453e98d-c621-4498-ad81-8cbc7eee10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:08.672167 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.670545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:08.672167 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.670607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:08.672167 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.670681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:08.672167 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.670795 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:08.672167 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.670860 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:08.672167 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:08.670929 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:08.809896 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.809858 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"52755e09a034ef6ecd3fb1e983ec8928085eb492fb38d386decbd7c2a3da33c1"} Nov 02 22:05:08.810088 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.809947 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"c0d0f76e61bf25699990005e0c2fb11646f278d0fad8ba5412718db1cf3a017a"} Nov 02 22:05:08.810088 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.809987 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"c6763eb2e7d80d79b82b708ba53716797bbf7e4866a01e9523f600e0a9d7800d"} Nov 02 22:05:08.810088 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.810014 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"995a829bda1592e50b80d50f03655d2c99cd107795f5999c594910b5dc7cdf31"} Nov 02 22:05:08.810088 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.810041 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"73ab0cfa6292c5229312b6ea30279b27c5a125b91645935b00ff5ba09d4cbf57"} Nov 02 22:05:08.810088 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:08.810068 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"c7a63dada19b619e5d5df1c08c70a07c3b81dfe8a50862bd0a2558013fae2c4d"} Nov 02 22:05:09.815268 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:09.815236 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="d857fa2ca218e11ac36e8cc7447e5ce64d018889adc59564c059caf8fa9cff12" exitCode=0 Nov 02 22:05:09.815762 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:09.815291 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerDied","Data":"d857fa2ca218e11ac36e8cc7447e5ce64d018889adc59564c059caf8fa9cff12"} Nov 02 22:05:10.673480 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:10.673428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:10.673480 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:10.673478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:10.673728 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:10.673538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:10.673728 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:10.673556 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:10.673728 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:10.673652 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:10.673929 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:10.673738 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:10.821493 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:10.821461 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="258c146ecb32a4d929348378f40c15006c61311707cf2442206831e3b383adb5" exitCode=0 Nov 02 22:05:10.821978 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:10.821507 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerDied","Data":"258c146ecb32a4d929348378f40c15006c61311707cf2442206831e3b383adb5"} Nov 02 22:05:11.826763 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:11.826723 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"ce86ede657139feb82f9b93ed415ca73aca48fa222f0102b58b37443df9070d7"} Nov 02 22:05:11.829893 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:11.829861 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-55xk7" event={"ID":"ed940156-2f94-4372-a949-f07e265f0c8b","Type":"ContainerStarted","Data":"4b731fb32117f7352f7e06ae84fae96c9a26402b0807f0056d60c0f08e41497e"} Nov 02 22:05:11.956046 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:11.955991 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-55xk7" podStartSLOduration=9.008831853 podStartE2EDuration="21.955976518s" totalImagesPullingTime="12.947144665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:11.955949015 +0000 UTC m=+21.784788153" watchObservedRunningTime="2026-11-02 22:05:11.955976518 +0000 UTC m=+21.784815650" Nov 02 22:05:12.673843 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:12.673814 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:12.673843 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:12.673833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:12.673843 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:12.673835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:12.674130 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:12.673933 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:12.674130 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:12.673978 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:12.674130 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:12.674052 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:13.838624 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:13.838415 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" event={"ID":"d6ed1d10-fcee-4a00-a544-efb9346dac66","Type":"ContainerStarted","Data":"ec4f26ac33e441e56c508f9625a2569709187e0e8893320ef5ce040988bdad0b"} Nov 02 22:05:13.839070 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:13.838696 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:05:13.855538 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:13.855508 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:05:14.125380 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:14.125263 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" podStartSLOduration=8.912272704 podStartE2EDuration="24.125250893s" totalImagesPullingTime="15.212978189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:14.125021217 +0000 UTC m=+23.953860369" watchObservedRunningTime="2026-11-02 22:05:14.125250893 +0000 UTC m=+23.954090031" Nov 02 22:05:14.670659 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:14.670611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:14.670881 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:14.670683 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:14.670881 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:14.670703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:14.670881 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:14.670821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:14.671041 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:14.670872 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:14.671041 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:14.670942 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:14.841158 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:14.841123 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:05:14.841626 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:14.841191 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:05:14.856786 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:14.856758 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:05:16.671107 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:16.671077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:16.671107 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:16.671089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:16.671550 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:16.671237 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:16.671550 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:16.671269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:16.671550 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:16.671344 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:16.671550 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:16.671416 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:18.673680 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:18.673646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:18.674088 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:18.673744 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:18.674088 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:18.673753 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:18.674088 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:18.673837 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:18.674088 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:18.673754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:18.674088 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:18.673919 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:20.671000 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:20.670963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:20.671382 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:20.671053 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:20.671382 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:20.671134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:20.671382 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:20.671225 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:20.671382 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:20.671280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:20.671382 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:20.671349 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:22.670910 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:22.670872 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:22.670910 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:22.670894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:22.671460 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:22.670876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:22.671460 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:22.670983 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:22.671460 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:22.671029 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:22.671460 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:22.671116 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:24.557766 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:24.557728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:24.558237 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:24.557800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:24.558237 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.557898 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:24.558237 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.557898 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:24.558237 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.557965 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs podName:6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:56.557947116 +0000 UTC m=+66.386786251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs") pod "network-metrics-daemon-h7lsc" (UID: "6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:24.558237 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.557979 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret podName:74da9552-2470-4a9f-8160-d3a29aa5d9f5 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:56.557973359 +0000 UTC m=+66.386812488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret") pod "global-pull-secret-syncer-knp2v" (UID: "74da9552-2470-4a9f-8160-d3a29aa5d9f5") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:24.658060 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:24.658028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:24.658212 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.658190 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:24.658212 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.658206 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:24.658290 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.658219 2574 projected.go:196] Error preparing data for projected volume kube-api-access-kwd52 for pod openshift-network-diagnostics/network-check-target-pb5sz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:24.658290 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.658277 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52 podName:e453e98d-c621-4498-ad81-8cbc7eee10fe nodeName:}" failed. No retries permitted until 2026-11-02 22:05:56.658262802 +0000 UTC m=+66.487101919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwd52" (UniqueName: "kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52") pod "network-check-target-pb5sz" (UID: "e453e98d-c621-4498-ad81-8cbc7eee10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:24.670223 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:24.670207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:24.670272 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:24.670224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:24.670315 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:24.670303 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:24.670347 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.670302 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:24.670408 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.670379 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:24.670461 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:24.670444 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:25.746787 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:25.746753 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-knp2v"] Nov 02 22:05:25.747286 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:25.746878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:25.747286 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:25.746998 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:25.747501 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:25.747481 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7lsc"] Nov 02 22:05:25.747612 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:25.747598 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:25.747727 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:25.747708 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:25.748054 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:25.748034 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pb5sz"] Nov 02 22:05:25.748163 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:25.748116 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:25.748223 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:25.748209 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:27.670215 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:27.670178 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:27.670612 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:27.670289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:27.670612 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:27.670313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:27.670612 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:27.670305 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:27.670612 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:27.670398 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:27.670612 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:27.670480 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:29.670199 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:29.670166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:29.670716 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:29.670244 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:29.670716 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:29.670337 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:29.670716 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:29.670341 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:29.670716 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:29.670452 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:29.670716 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:29.670521 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:31.670319 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:31.670108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:31.670768 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:31.670126 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:31.670768 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:31.670416 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:31.670768 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:31.670169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:31.670768 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:31.670461 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:31.670768 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:31.670597 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:31.796288 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:31.796263 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6lwq6_22f84c42-8ee4-4133-a636-f02598d2b61b/dns-node-resolver/0.log" Nov 02 22:05:32.627877 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:32.627851 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lsbsf_cc3ac303-6d4c-4f0a-9279-2b3c428adccd/node-ca/0.log" Nov 02 22:05:33.670815 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:33.670778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:33.671221 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:33.670778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:33.671221 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:33.670894 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:33.671221 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:33.670981 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:33.671221 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:33.670779 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:33.671221 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:33.671055 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:35.670771 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:35.670680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:35.671148 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:35.670790 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:35.671148 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:35.670798 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:35.671148 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:35.670822 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:35.671148 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:35.670898 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:35.671148 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:35.670966 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:37.670730 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:37.670680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:37.670730 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:37.670727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:37.671253 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:37.670702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:37.671253 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:37.670810 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:37.671253 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:37.670873 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:37.671253 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:37.670954 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:39.670374 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:39.670331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:39.670794 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:39.670379 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:39.670794 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:39.670384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:39.670794 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:39.670454 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:39.670794 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:39.670546 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:39.670794 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:39.670631 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:41.671127 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:41.671045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:41.671127 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:41.671078 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:41.671127 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:41.671098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:41.671617 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:41.671193 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:41.671617 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:41.671227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:41.671617 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:41.671317 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:43.670608 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:43.670543 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:43.671101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:43.670620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:43.671101 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:43.670586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:43.671101 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:43.670698 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:43.671101 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:43.670775 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:43.671101 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:43.670843 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:45.670889 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:45.670856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:45.671298 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:45.670899 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:45.671298 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:45.670960 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:45.671298 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:45.670974 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7lsc" podUID="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" Nov 02 22:05:45.671298 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:45.671031 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb5sz" podUID="e453e98d-c621-4498-ad81-8cbc7eee10fe" Nov 02 22:05:45.671298 ip-10-0-143-75 kubenswrapper[2574]: E1102 22:05:45.671099 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-knp2v" podUID="74da9552-2470-4a9f-8160-d3a29aa5d9f5" Nov 02 22:05:46.427020 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.426991 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeReady" Nov 02 22:05:46.427184 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.427129 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 22:05:46.547111 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.547073 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vxpxh"] Nov 02 22:05:46.549384 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.549368 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8zm4n"] Nov 02 22:05:46.549509 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.549492 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.551351 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.551335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:46.599205 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.599174 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p27gn"] Nov 02 22:05:46.601427 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.601412 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vxpxh"] Nov 02 22:05:46.601526 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.601512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.604219 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.604191 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:05:46.604358 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.604241 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:05:46.604358 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.604241 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t4sw7\"" Nov 02 22:05:46.611278 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.608629 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:05:46.611278 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.610183 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p27gn"] Nov 02 22:05:46.659224 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.659192 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zm4n"] Nov 02 22:05:46.704760 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln994\" (UniqueName: \"kubernetes.io/projected/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-kube-api-access-ln994\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:46.704760 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704750 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ebd138c-3a98-4268-b843-7b18f6ab68f8-crio-socket\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704749 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704783 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704803 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ebd138c-3a98-4268-b843-7b18f6ab68f8-data-volume\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704837 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4gtmj\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-cert\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704888 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5sg6\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704934 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704971 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.704983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-config-volume\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.705016 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-tmp-dir\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.705069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzjj\" (UniqueName: \"kubernetes.io/projected/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-api-access-9jzjj\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.705122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-metrics-tls\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.705191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ljx\" (UniqueName: \"kubernetes.io/projected/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-kube-api-access-c2ljx\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.705279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.705227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ebd138c-3a98-4268-b843-7b18f6ab68f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.806468 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-tmp-dir\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.806468 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzjj\" (UniqueName: \"kubernetes.io/projected/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-api-access-9jzjj\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.806721 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-metrics-tls\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.806721 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ljx\" (UniqueName: \"kubernetes.io/projected/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-kube-api-access-c2ljx\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.806721 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ebd138c-3a98-4268-b843-7b18f6ab68f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.806721 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln994\" (UniqueName: \"kubernetes.io/projected/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-kube-api-access-ln994\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:46.806889 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ebd138c-3a98-4268-b843-7b18f6ab68f8-crio-socket\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.806889 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ebd138c-3a98-4268-b843-7b18f6ab68f8-data-volume\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.806889 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-cert\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:46.806889 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.806889 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-config-volume\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.807126 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806896 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-tmp-dir\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.807126 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.806927 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ebd138c-3a98-4268-b843-7b18f6ab68f8-crio-socket\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.807270 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.807241 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ebd138c-3a98-4268-b843-7b18f6ab68f8-data-volume\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.807472 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.807427 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-config-volume\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.807472 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.807458 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.810721 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.810696 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-metrics-tls\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.810817 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.810749 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ebd138c-3a98-4268-b843-7b18f6ab68f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.810988 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.810968 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-cert\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:46.855896 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.855869 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:05:46.870014 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.869981 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzjj\" (UniqueName: \"kubernetes.io/projected/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-api-access-9jzjj\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:46.870155 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.870086 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ljx\" (UniqueName: \"kubernetes.io/projected/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-kube-api-access-c2ljx\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:46.870279 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.870262 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln994\" (UniqueName: \"kubernetes.io/projected/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-kube-api-access-ln994\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:46.914792 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:46.914757 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:05:47.164016 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.163925 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:47.166909 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.166890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:05:47.302481 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.302458 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p27gn"] Nov 02 22:05:47.535732 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:05:47.535703 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c3a586_fa6f_42bd_a0d8_42211a9ae53a.slice/crio-034147492ccae2628bf3132c590351fbe991e7a1563f5cb23b5d2f474160b86e WatchSource:0}: Error finding container 034147492ccae2628bf3132c590351fbe991e7a1563f5cb23b5d2f474160b86e: Status 404 returned error can't find the container with id 034147492ccae2628bf3132c590351fbe991e7a1563f5cb23b5d2f474160b86e Nov 02 22:05:47.537787 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.537765 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vxpxh"] Nov 02 22:05:47.542147 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:05:47.542122 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb494cbbd_3f65_4fb9_8454_c3d3fd12a5d9.slice/crio-de3e1c9da2ae2411a61235138f346331941530d7cae56b39f4a6497f7af03ed7 WatchSource:0}: Error finding container de3e1c9da2ae2411a61235138f346331941530d7cae56b39f4a6497f7af03ed7: Status 404 returned error can't find the container with id de3e1c9da2ae2411a61235138f346331941530d7cae56b39f4a6497f7af03ed7 Nov 02 22:05:47.544233 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.544210 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zm4n"] Nov 02 22:05:47.670779 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.670738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:47.670938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.670842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:47.670938 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.670929 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:47.725658 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.725617 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:05:47.726076 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.725714 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6jxx8\"" Nov 02 22:05:47.726076 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.725799 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:05:47.726076 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.725813 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:05:47.726076 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.725835 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:05:47.726076 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.725837 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4knj\"" Nov 02 22:05:47.726076 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.725911 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g764w\"" Nov 02 22:05:47.915684 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.915588 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zm4n" event={"ID":"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9","Type":"ContainerStarted","Data":"de3e1c9da2ae2411a61235138f346331941530d7cae56b39f4a6497f7af03ed7"} Nov 02 22:05:47.917046 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.917022 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p27gn" event={"ID":"7ebd138c-3a98-4268-b843-7b18f6ab68f8","Type":"ContainerStarted","Data":"8a65bf1a7887a98f631e71180af25e2cf6abc421e5c6b275dc8b95e12e1e48ef"} Nov 02 22:05:47.917175 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.917074 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p27gn" event={"ID":"7ebd138c-3a98-4268-b843-7b18f6ab68f8","Type":"ContainerStarted","Data":"62e7c81fbbf8353a15294b18136e861f5526541c49215151926abc72e867d3b7"} Nov 02 22:05:47.918037 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:47.918013 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxpxh" event={"ID":"76c3a586-fa6f-42bd-a0d8-42211a9ae53a","Type":"ContainerStarted","Data":"034147492ccae2628bf3132c590351fbe991e7a1563f5cb23b5d2f474160b86e"} Nov 02 22:05:48.923080 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:48.923002 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p27gn" event={"ID":"7ebd138c-3a98-4268-b843-7b18f6ab68f8","Type":"ContainerStarted","Data":"25855c15d02505c2776527f65f8b11d51591c75bd14499df42d9371de83d8152"} Nov 02 22:05:49.946986 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:49.946951 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc"] Nov 02 22:05:49.949829 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:49.949800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.057597 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.057479 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:05:50.057597 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.057595 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 22:05:50.057817 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.057690 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nh6bm\"" Nov 02 22:05:50.058350 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.058328 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:05:50.063849 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.063818 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc"] Nov 02 22:05:50.109390 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.109358 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 22:05:50.109591 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.109417 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:05:50.131151 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.131112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.131355 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.131168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28191529-9de8-4283-bd79-6b4de730919e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.131355 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.131332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gvfm\" (UniqueName: \"kubernetes.io/projected/28191529-9de8-4283-bd79-6b4de730919e-kube-api-access-5gvfm\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.131461 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.131392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.232462 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.232376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.232462 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.232456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.232732 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.232483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28191529-9de8-4283-bd79-6b4de730919e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.232732 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.232542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvfm\" (UniqueName: \"kubernetes.io/projected/28191529-9de8-4283-bd79-6b4de730919e-kube-api-access-5gvfm\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.234660 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.234633 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28191529-9de8-4283-bd79-6b4de730919e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.235905 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.235873 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.237651 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.237625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.296935 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.296901 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvfm\" (UniqueName: \"kubernetes.io/projected/28191529-9de8-4283-bd79-6b4de730919e-kube-api-access-5gvfm\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.565132 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.565091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:05:50.930013 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.929974 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxpxh" event={"ID":"76c3a586-fa6f-42bd-a0d8-42211a9ae53a","Type":"ContainerStarted","Data":"ca25ee557e9b37e17decbec661060eaa7969eabe1c209c9aa2ba37ab9ea522b3"} Nov 02 22:05:50.930212 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.930039 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxpxh" event={"ID":"76c3a586-fa6f-42bd-a0d8-42211a9ae53a","Type":"ContainerStarted","Data":"951333ab5d775f69d8e74b9a8f1290561c2e7fdeac9b0c22b0c9f3149e081a0e"} Nov 02 22:05:50.930212 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.930069 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vxpxh" Nov 02 22:05:50.932376 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.932350 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p27gn" event={"ID":"7ebd138c-3a98-4268-b843-7b18f6ab68f8","Type":"ContainerStarted","Data":"c0f85b2e2fe69f4f8b9081204481530384e03aab3ae1fdae7e555114bdbf2549"} Nov 02 22:05:50.949745 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:50.949705 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc"] Nov 02 22:05:50.975527 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:05:50.975498 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28191529_9de8_4283_bd79_6b4de730919e.slice/crio-cda3330bd7923fa858ca326092dc5919a2888505c1da96d4d66dbe4677ab8247 WatchSource:0}: Error finding container cda3330bd7923fa858ca326092dc5919a2888505c1da96d4d66dbe4677ab8247: Status 404 returned error can't find the container with id cda3330bd7923fa858ca326092dc5919a2888505c1da96d4d66dbe4677ab8247 Nov 02 22:05:51.052510 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:51.052454 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vxpxh" podStartSLOduration=2.620129886 podStartE2EDuration="5.052435121s" totalImagesPullingTime="2.432305235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:51.051256906 +0000 UTC m=+60.880096056" watchObservedRunningTime="2026-11-02 22:05:51.052435121 +0000 UTC m=+60.881274261" Nov 02 22:05:51.174135 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:51.174045 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p27gn" podStartSLOduration=1.781917623 podStartE2EDuration="5.174029303s" totalImagesPullingTime="3.39211168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:51.173634087 +0000 UTC m=+61.002473225" watchObservedRunningTime="2026-11-02 22:05:51.174029303 +0000 UTC m=+61.002868441" Nov 02 22:05:51.936022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:51.935980 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" event={"ID":"28191529-9de8-4283-bd79-6b4de730919e","Type":"ContainerStarted","Data":"cda3330bd7923fa858ca326092dc5919a2888505c1da96d4d66dbe4677ab8247"} Nov 02 22:05:51.937396 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:51.937369 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zm4n" event={"ID":"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9","Type":"ContainerStarted","Data":"a5cc4e305dfa6b1d8ad68567eb1660189525af854e52f87de946f2d14fdce85e"} Nov 02 22:05:52.061075 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:52.061019 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8zm4n" podStartSLOduration=2.397846484 podStartE2EDuration="6.061004747s" totalImagesPullingTime="3.663158263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:52.060649615 +0000 UTC m=+61.889488753" watchObservedRunningTime="2026-11-02 22:05:52.061004747 +0000 UTC m=+61.889843884" Nov 02 22:05:53.944549 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:53.944505 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" event={"ID":"28191529-9de8-4283-bd79-6b4de730919e","Type":"ContainerStarted","Data":"36678a18cafe423fb3737cedb0c41712864c8119857aa24ddbbe5953d71aa660"} Nov 02 22:05:53.944954 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:53.944593 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" event={"ID":"28191529-9de8-4283-bd79-6b4de730919e","Type":"ContainerStarted","Data":"540316299af3c1a58657e692b0d7e435355fa6a9229b897a99cc28d73ab50711"} Nov 02 22:05:54.066624 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:54.066554 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" podStartSLOduration=2.915960101 podStartE2EDuration="5.066541201s" totalImagesPullingTime="2.1505811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:54.066157534 +0000 UTC m=+63.894996671" watchObservedRunningTime="2026-11-02 22:05:54.066541201 +0000 UTC m=+63.895380338" Nov 02 22:05:56.585244 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.585200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:56.585244 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.585252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:56.685737 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.685701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:56.690770 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.690743 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:05:56.690770 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.690746 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:05:56.698351 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.698320 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:56.698475 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.698333 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:56.739262 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.739234 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:05:56.799389 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.799355 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:05:56.809091 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:56.809066 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:57.035628 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.035601 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g764w\"" Nov 02 22:05:57.041513 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.041495 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4knj\"" Nov 02 22:05:57.042598 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.042561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:05:57.048874 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.048857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:05:57.062775 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.062736 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6jxx8\"" Nov 02 22:05:57.073021 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.072948 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:05:57.410333 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.410233 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pb5sz"] Nov 02 22:05:57.443393 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:05:57.443367 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode453e98d_c621_4498_ad81_8cbc7eee10fe.slice/crio-8a8b00d56db228da808fa1c26314ce2d562f10d6a197c620fc47845e90d6e881 WatchSource:0}: Error finding container 8a8b00d56db228da808fa1c26314ce2d562f10d6a197c620fc47845e90d6e881: Status 404 returned error can't find the container with id 8a8b00d56db228da808fa1c26314ce2d562f10d6a197c620fc47845e90d6e881 Nov 02 22:05:57.454259 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:05:57.454219 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74da9552_2470_4a9f_8160_d3a29aa5d9f5.slice/crio-709bb7207c08474bd8a2b604c828381882bafe8fa2c37dd8dc81149905f36435 WatchSource:0}: Error finding container 709bb7207c08474bd8a2b604c828381882bafe8fa2c37dd8dc81149905f36435: Status 404 returned error can't find the container with id 709bb7207c08474bd8a2b604c828381882bafe8fa2c37dd8dc81149905f36435 Nov 02 22:05:57.456152 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.456125 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-knp2v"] Nov 02 22:05:57.457327 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:05:57.457304 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6636a0f1_0b17_4bcd_9a9b_6d73e5c558d8.slice/crio-9c8a543704dd23caa16fecdecdc566ec9410669a2f09c6196ed5a0da8621d9d0 WatchSource:0}: Error finding container 9c8a543704dd23caa16fecdecdc566ec9410669a2f09c6196ed5a0da8621d9d0: Status 404 returned error can't find the container with id 9c8a543704dd23caa16fecdecdc566ec9410669a2f09c6196ed5a0da8621d9d0 Nov 02 22:05:57.459105 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.459087 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7lsc"] Nov 02 22:05:57.605921 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.605884 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7sslx"] Nov 02 22:05:57.609012 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.608995 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.685527 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.685420 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:05:57.685527 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.685437 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:05:57.685723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.685616 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5dnx\"" Nov 02 22:05:57.693981 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.693958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7x7\" (UniqueName: \"kubernetes.io/projected/54880730-d3aa-4d76-8234-0dc766363113-kube-api-access-pc7x7\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694109 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.693987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694109 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.694024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-textfile\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694177 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.694105 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-root\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694177 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.694128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-wtmp\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694177 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.694145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-metrics-client-ca\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694271 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.694204 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694271 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.694244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-sys\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.694271 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.694267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-tls\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.705776 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.705749 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:05:57.794790 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-sys\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.794790 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-tls\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7x7\" (UniqueName: \"kubernetes.io/projected/54880730-d3aa-4d76-8234-0dc766363113-kube-api-access-pc7x7\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794857 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-sys\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-textfile\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-root\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-wtmp\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-metrics-client-ca\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795022 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.794987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795319 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.795034 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-root\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795319 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.795157 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-wtmp\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795388 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.795343 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-textfile\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795503 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.795485 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.795600 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.795501 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-metrics-client-ca\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.797373 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.797349 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.797502 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.797489 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-tls\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.857833 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.857802 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7x7\" (UniqueName: \"kubernetes.io/projected/54880730-d3aa-4d76-8234-0dc766363113-kube-api-access-pc7x7\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.922485 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.922447 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:05:57.932552 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:05:57.932517 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54880730_d3aa_4d76_8234_0dc766363113.slice/crio-b0d9566f1ba5727be81ce53a4d678b4e57d6593a37f89305f736482e2ae42a5d WatchSource:0}: Error finding container b0d9566f1ba5727be81ce53a4d678b4e57d6593a37f89305f736482e2ae42a5d: Status 404 returned error can't find the container with id b0d9566f1ba5727be81ce53a4d678b4e57d6593a37f89305f736482e2ae42a5d Nov 02 22:05:57.961953 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.961916 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sslx" event={"ID":"54880730-d3aa-4d76-8234-0dc766363113","Type":"ContainerStarted","Data":"b0d9566f1ba5727be81ce53a4d678b4e57d6593a37f89305f736482e2ae42a5d"} Nov 02 22:05:57.963015 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.962981 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7lsc" event={"ID":"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8","Type":"ContainerStarted","Data":"9c8a543704dd23caa16fecdecdc566ec9410669a2f09c6196ed5a0da8621d9d0"} Nov 02 22:05:57.964156 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.964127 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pb5sz" event={"ID":"e453e98d-c621-4498-ad81-8cbc7eee10fe","Type":"ContainerStarted","Data":"8a8b00d56db228da808fa1c26314ce2d562f10d6a197c620fc47845e90d6e881"} Nov 02 22:05:57.965095 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:57.965075 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-knp2v" event={"ID":"74da9552-2470-4a9f-8160-d3a29aa5d9f5","Type":"ContainerStarted","Data":"709bb7207c08474bd8a2b604c828381882bafe8fa2c37dd8dc81149905f36435"} Nov 02 22:05:59.976436 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:59.976387 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sslx" event={"ID":"54880730-d3aa-4d76-8234-0dc766363113","Type":"ContainerStarted","Data":"c6dbecbc591073c45860f43d95775f323423dd4cb7e3339348364657aabfccb3"} Nov 02 22:05:59.979247 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:59.979216 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7lsc" event={"ID":"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8","Type":"ContainerStarted","Data":"93acf89369241f1c062970039ffd7235281f55a2aa46ae8df46a59017782c968"} Nov 02 22:05:59.979408 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:05:59.979275 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7lsc" event={"ID":"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8","Type":"ContainerStarted","Data":"0acf16cb0a8ed207232bf0d1c14d27f130762a7389f54d6eea26d553e31379ea"} Nov 02 22:06:00.316492 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.316427 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h7lsc" podStartSLOduration=67.508613401 podStartE2EDuration="1m9.316405476s" totalImagesPullingTime="1.807792075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:00.22044319 +0000 UTC m=+70.049282329" watchObservedRunningTime="2026-11-02 22:06:00.316405476 +0000 UTC m=+70.145244616" Nov 02 22:06:00.316989 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.316964 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq"] Nov 02 22:06:00.320497 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.319804 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.374723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.374551 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 22:06:00.374723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.374593 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 22:06:00.374723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.374601 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ddaq9n2d3jel1\"" Nov 02 22:06:00.374723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.374634 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 22:06:00.374723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.374676 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zsfxd\"" Nov 02 22:06:00.374723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.374684 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 22:06:00.374723 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.374695 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 22:06:00.381683 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.381651 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq"] Nov 02 22:06:00.414843 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.414751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b6f934-07bb-42c1-8e3d-5bc2369b1716-metrics-client-ca\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.414843 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.414805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.415085 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.414891 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b6f934-07bb-42c1-8e3d-5bc2369b1716-tmp\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.415085 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.414927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-grpc-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.415085 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.415017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.415085 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.415046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.415085 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.415073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnqm\" (UniqueName: \"kubernetes.io/projected/63b6f934-07bb-42c1-8e3d-5bc2369b1716-kube-api-access-8cnqm\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.415315 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.415103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.415315 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.415192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516028 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.515976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516231 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b6f934-07bb-42c1-8e3d-5bc2369b1716-metrics-client-ca\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516231 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516231 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b6f934-07bb-42c1-8e3d-5bc2369b1716-tmp\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516231 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-grpc-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516231 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516231 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516475 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnqm\" (UniqueName: \"kubernetes.io/projected/63b6f934-07bb-42c1-8e3d-5bc2369b1716-kube-api-access-8cnqm\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.516475 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.517235 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.516913 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b6f934-07bb-42c1-8e3d-5bc2369b1716-metrics-client-ca\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.517235 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.517194 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b6f934-07bb-42c1-8e3d-5bc2369b1716-tmp\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.519890 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.519854 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.521206 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.521183 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.521740 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.521715 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-grpc-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.521822 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.521771 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.521822 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.521774 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.523185 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.523162 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.580898 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.580803 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnqm\" (UniqueName: \"kubernetes.io/projected/63b6f934-07bb-42c1-8e3d-5bc2369b1716-kube-api-access-8cnqm\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.641860 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.641826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:00.939827 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.939755 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:00.986431 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.986395 2574 generic.go:356] "Generic (PLEG): container finished" podID="54880730-d3aa-4d76-8234-0dc766363113" containerID="c6dbecbc591073c45860f43d95775f323423dd4cb7e3339348364657aabfccb3" exitCode=0 Nov 02 22:06:00.987478 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:00.987452 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sslx" event={"ID":"54880730-d3aa-4d76-8234-0dc766363113","Type":"ContainerDied","Data":"c6dbecbc591073c45860f43d95775f323423dd4cb7e3339348364657aabfccb3"} Nov 02 22:06:01.119002 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:01.118970 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq"] Nov 02 22:06:01.119640 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:06:01.119604 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b6f934_07bb_42c1_8e3d_5bc2369b1716.slice/crio-602b835d9710f35a787b961fe64a02e27a92d524182584a8594ad05eee562bb1 WatchSource:0}: Error finding container 602b835d9710f35a787b961fe64a02e27a92d524182584a8594ad05eee562bb1: Status 404 returned error can't find the container with id 602b835d9710f35a787b961fe64a02e27a92d524182584a8594ad05eee562bb1 Nov 02 22:06:01.992808 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:01.992710 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pb5sz" event={"ID":"e453e98d-c621-4498-ad81-8cbc7eee10fe","Type":"ContainerStarted","Data":"8e2238fd7d469bb11c0c7431b27f590f10506c28338ab7bf371903150a0d08b2"} Nov 02 22:06:01.993561 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:01.993537 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:06:01.996376 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:01.996285 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sslx" event={"ID":"54880730-d3aa-4d76-8234-0dc766363113","Type":"ContainerStarted","Data":"fd3a322237b8921b38babc3dfb90b8dc97c1a6f4af7bc16d92d6cd143d085aad"} Nov 02 22:06:01.996376 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:01.996340 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sslx" event={"ID":"54880730-d3aa-4d76-8234-0dc766363113","Type":"ContainerStarted","Data":"8613f1ce45ba21021868d306f77037e53165022cc4a79e8185e27e5e2f6ee47b"} Nov 02 22:06:01.997696 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:01.997661 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" event={"ID":"63b6f934-07bb-42c1-8e3d-5bc2369b1716","Type":"ContainerStarted","Data":"602b835d9710f35a787b961fe64a02e27a92d524182584a8594ad05eee562bb1"} Nov 02 22:06:02.114728 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.114674 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pb5sz" podStartSLOduration=67.891816339 podStartE2EDuration="1m12.114653257s" totalImagesPullingTime="4.222836918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:02.114061728 +0000 UTC m=+71.942900868" watchObservedRunningTime="2026-11-02 22:06:02.114653257 +0000 UTC m=+71.943492394" Nov 02 22:06:02.241910 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.241849 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7sslx" podStartSLOduration=3.27333275 podStartE2EDuration="5.241830105s" totalImagesPullingTime="1.875753557s" totalInitContainerRuntime="92.743798ms" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:02.240592921 +0000 UTC m=+72.069432067" watchObservedRunningTime="2026-11-02 22:06:02.241830105 +0000 UTC m=+72.070669244" Nov 02 22:06:02.353719 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.353683 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw"] Nov 02 22:06:02.356684 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.356652 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:02.411856 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.411817 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 22:06:02.412019 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.411899 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wsgcd\"" Nov 02 22:06:02.413134 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.413106 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw"] Nov 02 22:06:02.430512 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.430469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6gk\" (UniqueName: \"kubernetes.io/projected/efae3ac8-4b9e-4598-99b0-33eb10cd1935-kube-api-access-sk6gk\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:02.430700 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.430531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efae3ac8-4b9e-4598-99b0-33eb10cd1935-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:02.531950 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.531906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6gk\" (UniqueName: \"kubernetes.io/projected/efae3ac8-4b9e-4598-99b0-33eb10cd1935-kube-api-access-sk6gk\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:02.532127 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.531976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efae3ac8-4b9e-4598-99b0-33eb10cd1935-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:02.535109 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.535073 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efae3ac8-4b9e-4598-99b0-33eb10cd1935-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:02.600978 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.600949 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6gk\" (UniqueName: \"kubernetes.io/projected/efae3ac8-4b9e-4598-99b0-33eb10cd1935-kube-api-access-sk6gk\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:02.670912 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:02.670820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:03.050705 ip-10-0-143-75 kubenswrapper[2574]: W1102 22:06:03.050670 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefae3ac8_4b9e_4598_99b0_33eb10cd1935.slice/crio-9c686753802cdb04ce0dc0a473dc992383809915f411027a2f480c484adab164 WatchSource:0}: Error finding container 9c686753802cdb04ce0dc0a473dc992383809915f411027a2f480c484adab164: Status 404 returned error can't find the container with id 9c686753802cdb04ce0dc0a473dc992383809915f411027a2f480c484adab164 Nov 02 22:06:03.052387 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.052361 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw"] Nov 02 22:06:03.374383 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.374334 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-674c5d5fcb-845r7"] Nov 02 22:06:03.377930 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.377900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.432346 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.432308 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 22:06:03.432680 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.432654 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 22:06:03.432788 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.432715 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8d44q\"" Nov 02 22:06:03.432788 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.432741 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 22:06:03.432951 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.432937 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 22:06:03.439914 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.439877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.440040 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.439935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-serving-certs-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.440040 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.439975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgtj\" (UniqueName: \"kubernetes.io/projected/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-kube-api-access-vlgtj\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.440040 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.440010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-federate-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.440040 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.440036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.440242 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.440070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.440242 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.440103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-metrics-client-ca\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.440242 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.440134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.451672 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.451618 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 22:06:03.505339 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.505306 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 22:06:03.538165 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.538135 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-674c5d5fcb-845r7"] Nov 02 22:06:03.541227 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgtj\" (UniqueName: \"kubernetes.io/projected/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-kube-api-access-vlgtj\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.541413 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-federate-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.541413 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.541413 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.541413 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-metrics-client-ca\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.541413 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.541707 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.541707 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.541449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-serving-certs-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.543201 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.542229 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-serving-certs-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.543447 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.543418 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-metrics-client-ca\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.544736 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.544706 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.545528 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.545505 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-federate-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.546277 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.546251 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.546543 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.546521 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.546833 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.546805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.606465 ip-10-0-143-75 kubenswrapper[2574]: I1102 22:06:03.606390 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgtj\" (UniqueName: \"kubernetes.io/projected/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-kube-api-access-vlgtj\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:03.669752 ip-10-0-143-75 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 22:06:03.682165 ip-10-0-143-75 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 22:06:03.682372 ip-10-0-143-75 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 22:06:03.682428 ip-10-0-143-75 systemd[1]: kubelet.service: Consumed 3.894s CPU time. Nov 02 22:06:03.690410 ip-10-0-143-75 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:06:03.773970 ip-10-0-143-75 kubenswrapper[5109]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:06:03.773970 ip-10-0-143-75 kubenswrapper[5109]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:06:03.773970 ip-10-0-143-75 kubenswrapper[5109]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:06:03.774852 ip-10-0-143-75 kubenswrapper[5109]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:06:03.777273 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777149 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.777273 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777270 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.777273 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777277 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777282 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777288 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777293 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777297 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777302 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777306 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777311 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777316 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777320 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777325 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777329 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777332 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777335 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777337 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777340 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777349 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777352 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777355 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.777450 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777358 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777360 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777363 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777365 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777369 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777371 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777374 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777377 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777379 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777382 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777385 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777387 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777390 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777392 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777396 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777399 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777402 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777404 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777407 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777409 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.777923 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777412 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777415 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777417 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777420 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777422 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777425 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777429 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777432 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777434 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777437 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777440 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777442 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777445 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777447 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777450 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777452 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777455 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777458 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777460 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777464 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.778445 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777466 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777469 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777471 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777474 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777477 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777479 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777483 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777485 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777488 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777493 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777497 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777499 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777502 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777505 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777507 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777511 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777514 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777516 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777520 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.779082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777523 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777526 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777529 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777531 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777534 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777536 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777539 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777542 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777544 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777547 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777549 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777552 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777554 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777557 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777559 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777562 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777580 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777583 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777586 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777590 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.779760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777592 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777595 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777598 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777601 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777604 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.777607 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778463 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778472 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778475 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778478 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778481 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778484 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778486 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778491 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778494 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778496 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778499 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778501 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778504 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.780476 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778506 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778509 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778514 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778517 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778520 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778522 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778525 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778527 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778530 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778533 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778535 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778538 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778541 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778543 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778546 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778548 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778551 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778554 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778556 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778561 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.781148 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778581 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778584 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778587 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778590 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778592 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778595 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778597 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778605 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778607 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778610 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778613 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778615 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778617 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778620 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778623 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778625 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778629 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778631 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778634 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778636 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.781908 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778639 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778641 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778644 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778646 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778649 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778652 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778654 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778657 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778660 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778662 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778666 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778673 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778678 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778682 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778687 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778691 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778694 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778698 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778702 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.782587 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778707 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778714 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778719 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778723 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778728 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778732 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778736 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778739 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778742 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778746 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778750 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778754 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778758 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778762 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778766 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778769 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778773 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778780 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778785 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.783252 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778790 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778797 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778801 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778806 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778810 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778814 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778817 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778821 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778825 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778829 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778833 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778837 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778842 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778847 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.778851 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779587 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779600 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779604 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779609 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.783931 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779613 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779617 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779622 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779626 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779630 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779635 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779638 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779640 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779643 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779645 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779649 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779652 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779655 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779659 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779664 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779669 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779673 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779676 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779679 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779682 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.784530 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779685 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779687 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779690 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779693 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779697 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779701 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779706 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779710 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779716 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779720 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779724 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779729 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779733 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779737 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779741 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779746 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779750 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779755 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779759 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.785258 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779764 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779769 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779774 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779778 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779786 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779793 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779798 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779803 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779807 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779812 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779817 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779821 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779826 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779831 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779836 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779841 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779847 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779851 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.785946 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779854 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779857 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779859 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779862 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779866 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779869 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779871 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779874 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779878 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779882 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779887 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779891 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779895 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779900 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779905 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779919 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779925 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779929 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779934 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779937 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.786578 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779940 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779943 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779947 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779952 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779956 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779961 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779966 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779971 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779975 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779980 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779984 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779986 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779989 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779992 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779994 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.779998 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780005 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780010 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780015 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780018 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.787295 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780021 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780023 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780026 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780028 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.780031 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780184 5109 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780205 5109 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780212 5109 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780219 5109 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780229 5109 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780234 5109 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780241 5109 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780248 5109 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780253 5109 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780258 5109 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780265 5109 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780270 5109 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780275 5109 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780278 5109 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780281 5109 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780284 5109 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780287 5109 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780290 5109 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780300 5109 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:06:03.788026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780305 5109 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780310 5109 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780315 5109 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780323 5109 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780330 5109 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780335 5109 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780341 5109 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780346 5109 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780353 5109 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780357 5109 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780362 5109 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780368 5109 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780373 5109 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780381 5109 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780387 5109 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780392 5109 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780397 5109 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780405 5109 flags.go:64] FLAG: --enable-server="true" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780410 5109 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780418 5109 flags.go:64] FLAG: --event-burst="100" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780424 5109 flags.go:64] FLAG: --event-qps="50" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780429 5109 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780434 5109 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780438 5109 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:06:03.788900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780442 5109 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780445 5109 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780450 5109 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780455 5109 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780460 5109 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780465 5109 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780469 5109 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780474 5109 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780479 5109 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780484 5109 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780488 5109 flags.go:64] FLAG: --feature-gates="" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780497 5109 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780502 5109 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780507 5109 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780512 5109 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780518 5109 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780523 5109 flags.go:64] FLAG: --help="false" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780527 5109 flags.go:64] FLAG: --hostname-override="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780532 5109 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780537 5109 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780541 5109 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780546 5109 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780552 5109 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:06:03.789797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780557 5109 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780562 5109 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780585 5109 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780591 5109 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780596 5109 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780601 5109 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780606 5109 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780611 5109 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780615 5109 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780620 5109 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780624 5109 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780628 5109 flags.go:64] FLAG: --lock-file="" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780633 5109 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780637 5109 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780640 5109 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780647 5109 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780650 5109 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780654 5109 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780656 5109 flags.go:64] FLAG: --logging-format="text" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780659 5109 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780664 5109 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780667 5109 flags.go:64] FLAG: --manifest-url="" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780670 5109 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780675 5109 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780679 5109 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:06:03.790632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780684 5109 flags.go:64] FLAG: --max-pods="110" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780687 5109 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780690 5109 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780693 5109 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780699 5109 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780704 5109 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780709 5109 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780713 5109 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780724 5109 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780731 5109 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780736 5109 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780741 5109 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780744 5109 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780747 5109 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780750 5109 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780755 5109 flags.go:64] FLAG: --port="10250" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780760 5109 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780765 5109 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-051bf643ca3f39370" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780770 5109 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780774 5109 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780779 5109 flags.go:64] FLAG: --register-node="true" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780783 5109 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780789 5109 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780794 5109 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780804 5109 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780808 5109 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:06:03.791520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780812 5109 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780816 5109 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780820 5109 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780823 5109 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780825 5109 flags.go:64] FLAG: --runonce="false" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780828 5109 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780832 5109 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780836 5109 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780839 5109 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780842 5109 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780847 5109 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780852 5109 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780858 5109 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780863 5109 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780868 5109 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780875 5109 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780878 5109 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780882 5109 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780885 5109 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780888 5109 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780894 5109 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780897 5109 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780900 5109 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780905 5109 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780908 5109 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:06:03.792446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780911 5109 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780914 5109 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780917 5109 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780922 5109 flags.go:64] FLAG: --v="2" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780929 5109 flags.go:64] FLAG: --version="false" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780937 5109 flags.go:64] FLAG: --vmodule="" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780944 5109 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.780948 5109 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781134 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781139 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781142 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781144 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781148 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781152 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781157 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781166 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781172 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781175 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781178 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781180 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781183 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.793365 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781185 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781189 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781192 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781195 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781198 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781200 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781203 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781207 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781211 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781216 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781221 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781225 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781229 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781233 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781239 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781244 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781248 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781253 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781257 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781262 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.794339 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781264 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781267 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781269 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781273 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781278 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781282 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781286 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781291 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781294 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781297 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781300 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781303 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781305 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781309 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781312 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781316 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781320 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781325 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781330 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781334 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.795062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781339 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781343 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781348 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781352 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781357 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781362 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781367 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781371 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781376 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781378 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781382 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781388 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781396 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781403 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781409 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781412 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781415 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781418 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781420 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.795773 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781423 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781427 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781430 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781433 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781437 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781442 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781447 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781452 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781457 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781462 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781467 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781472 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781476 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781480 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781482 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781485 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781488 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781491 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781493 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781496 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.796424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781498 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781501 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781504 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781506 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781510 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781513 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781519 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781525 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781530 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781534 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781539 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781543 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781548 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781553 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.781620 5109 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:06:03.797136 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781689 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781696 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781701 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781709 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781714 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781718 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781723 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781727 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781731 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781736 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781741 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781745 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781749 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781754 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781758 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781762 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781767 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781771 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781776 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781781 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.797760 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781785 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781788 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781794 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781796 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781799 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781802 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781804 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781807 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781812 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781816 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781820 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781825 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781829 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781831 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781836 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781842 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781846 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781851 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781856 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.798494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781861 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781866 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781871 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781876 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781880 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781884 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781889 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781919 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781927 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781932 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781936 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781939 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781944 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781950 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781956 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781965 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781970 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781975 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781979 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.799173 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781983 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781988 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781993 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.781997 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782002 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782006 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782010 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782015 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782021 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782028 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782031 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782034 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782047 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782053 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782058 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782062 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782065 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782067 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782070 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782073 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.799925 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782075 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782078 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782081 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782083 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782088 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782092 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782099 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782105 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782114 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782119 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782122 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782127 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782132 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782137 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782141 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782146 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782150 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782155 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782161 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.800763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782166 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782171 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782177 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782181 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782185 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782189 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782193 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782198 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.782202 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.788389 5109 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.788416 5109 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.788439 5109 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.788451 5109 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788543 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788551 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788556 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788560 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788581 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788585 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.801482 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788590 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788594 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788599 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788603 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788607 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788611 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788618 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788625 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788630 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788635 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788640 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788644 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788649 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788653 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788657 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788662 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788666 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788671 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788675 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.802178 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788679 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788683 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788687 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788691 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788705 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788710 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788714 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788718 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788722 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788726 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788729 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788735 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788739 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788743 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788747 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788752 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788756 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788760 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788764 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788768 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.802951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788772 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788776 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788780 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788784 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788788 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788793 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788798 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788802 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788807 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788811 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788815 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788819 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788823 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788827 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788831 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788835 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788839 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788852 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788857 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.803705 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788860 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788864 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788869 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788873 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788878 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788883 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788888 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788892 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788896 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788900 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788904 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788908 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788912 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788916 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788920 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788924 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788929 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788933 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788937 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.804336 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788941 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788945 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788949 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788954 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788959 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788965 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788971 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788975 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788979 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788984 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788988 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.788992 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789002 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789006 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789011 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789014 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789019 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789023 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789028 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789033 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.804989 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789037 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789041 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789045 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.789088 5109 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789366 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789374 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789380 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789384 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789389 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789393 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789397 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789401 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789405 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789409 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789414 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789419 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.805666 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789424 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789429 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789435 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789441 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789446 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789450 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789454 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789458 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789462 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789473 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789478 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789482 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789486 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789491 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789496 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789500 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789504 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789508 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789512 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.806228 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789517 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789545 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789551 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789556 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789560 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789582 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789588 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789592 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789596 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789600 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789604 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789608 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789612 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789619 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789625 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789630 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789634 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789638 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789642 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.806920 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789646 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789650 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789655 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789659 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789670 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789675 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789679 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789683 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789687 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789691 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789695 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789699 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789703 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789707 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789711 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789715 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789719 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789723 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789727 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789730 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.807649 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789734 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789738 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789742 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789746 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789750 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789754 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789767 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789774 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789778 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789782 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789787 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789792 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789796 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789801 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789805 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789809 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789813 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789824 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789829 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789833 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.808332 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789837 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789841 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789845 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789849 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789853 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789857 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789862 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789867 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789871 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789875 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789879 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789884 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789888 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789892 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789896 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789902 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.809046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.789939 5109 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.789999 5109 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790006 5109 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790010 5109 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790016 5109 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790020 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790025 5109 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790029 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790033 5109 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790037 5109 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790041 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790045 5109 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790049 5109 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790053 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790063 5109 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790068 5109 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790072 5109 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790075 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790079 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:03.809808 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790083 5109 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790087 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790091 5109 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790096 5109 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790099 5109 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790103 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790107 5109 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790111 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790115 5109 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790119 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790122 5109 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790124 5109 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790128 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790132 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790136 5109 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790140 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790144 5109 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790149 5109 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790153 5109 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:03.810515 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790157 5109 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790161 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790164 5109 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790168 5109 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790172 5109 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790176 5109 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790180 5109 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790184 5109 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790186 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790194 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790197 5109 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790200 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790204 5109 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790208 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790212 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790215 5109 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790219 5109 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790223 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790227 5109 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:03.811345 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790231 5109 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790235 5109 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790239 5109 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790243 5109 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790247 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790251 5109 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790256 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790260 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790263 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790267 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790270 5109 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790275 5109 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790279 5109 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790283 5109 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790286 5109 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790291 5109 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790295 5109 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790303 5109 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790307 5109 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:03.812135 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790312 5109 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790315 5109 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790319 5109 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790322 5109 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790338 5109 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790342 5109 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790346 5109 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790350 5109 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790353 5109 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790357 5109 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790361 5109 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790364 5109 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790368 5109 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790371 5109 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790375 5109 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790379 5109 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790383 5109 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790387 5109 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790391 5109 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790396 5109 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:03.812901 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790400 5109 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790403 5109 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790407 5109 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790410 5109 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790415 5109 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790419 5109 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790423 5109 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790427 5109 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790430 5109 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790434 5109 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:03.790437 5109 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.790535 5109 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.791966 5109 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.792050 5109 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.792378 5109 server.go:1014] "Starting client certificate rotation" Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.792515 5109 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.792588 5109 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:59:50 +0000 UTC" deadline="2028-06-16 10:46:24.255980533 +0000 UTC" Nov 02 22:06:03.813620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.792640 5109 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14196h40m20.463346102s" Nov 02 22:06:03.814281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.793092 5109 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:06:03.814281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.798189 5109 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:06:03.814281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.799907 5109 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:06:03.814281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.802171 5109 log.go:25] "Validated CRI v1 image API" Nov 02 22:06:03.814281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.803288 5109 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:06:03.814281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.805584 5109 fs.go:135] Filesystem UUIDs: map[466778fa-b7a7-4ff6-be98-1b6516199dc5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a0445363-335d-4c8f-8c60-63c9a8e95090:/dev/nvme0n1p3] Nov 02 22:06:03.814540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.805618 5109 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/034147492ccae2628bf3132c590351fbe991e7a1563f5cb23b5d2f474160b86e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/034147492ccae2628bf3132c590351fbe991e7a1563f5cb23b5d2f474160b86e/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/051c1af1375591ce8b9003110ee894ae74962a0d86f33ded706a979592c68342/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/051c1af1375591ce8b9003110ee894ae74962a0d86f33ded706a979592c68342/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e2290f26a1c7b0a1235abbf0965ef1f5bb30cf0a16f024547401fb76a29890b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e2290f26a1c7b0a1235abbf0965ef1f5bb30cf0a16f024547401fb76a29890b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/602b835d9710f35a787b961fe64a02e27a92d524182584a8594ad05eee562bb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/602b835d9710f35a787b961fe64a02e27a92d524182584a8594ad05eee562bb1/userdata/shm major:0 minor:308 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60d62e41334654822970b4cda017d2643cf56db8869f668311d8826c21bbabd6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60d62e41334654822970b4cda017d2643cf56db8869f668311d8826c21bbabd6/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62e7c81fbbf8353a15294b18136e861f5526541c49215151926abc72e867d3b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62e7c81fbbf8353a15294b18136e861f5526541c49215151926abc72e867d3b7/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/709bb7207c08474bd8a2b604c828381882bafe8fa2c37dd8dc81149905f36435/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/709bb7207c08474bd8a2b604c828381882bafe8fa2c37dd8dc81149905f36435/userdata/shm major:0 minor:275 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89e7a1e5f423c963456d66b6293e7c2c5220df7af479ca412e548dc1205b2fcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89e7a1e5f423c963456d66b6293e7c2c5220df7af479ca412e548dc1205b2fcd/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a8b00d56db228da808fa1c26314ce2d562f10d6a197c620fc47845e90d6e881/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a8b00d56db228da808fa1c26314ce2d562f10d6a197c620fc47845e90d6e881/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c686753802cdb04ce0dc0a473dc992383809915f411027a2f480c484adab164/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c686753802cdb04ce0dc0a473dc992383809915f411027a2f480c484adab164/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c8a543704dd23caa16fecdecdc566ec9410669a2f09c6196ed5a0da8621d9d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c8a543704dd23caa16fecdecdc566ec9410669a2f09c6196ed5a0da8621d9d0/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0d9566f1ba5727be81ce53a4d678b4e57d6593a37f89305f736482e2ae42a5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0d9566f1ba5727be81ce53a4d678b4e57d6593a37f89305f736482e2ae42a5d/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5303ac1ad52ac0bfe694e1759e2af566d7b4d5bb4d877ece111dd9c71b9ee98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5303ac1ad52ac0bfe694e1759e2af566d7b4d5bb4d877ece111dd9c71b9ee98/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd19933d1a573d9cdaec74f596f7fa7384455d80c4b43f431c28ef0bee62ac0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd19933d1a573d9cdaec74f596f7fa7384455d80c4b43f431c28ef0bee62ac0e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cda3330bd7923fa858ca326092dc5919a2888505c1da96d4d66dbe4677ab8247/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cda3330bd7923fa858ca326092dc5919a2888505c1da96d4d66dbe4677ab8247/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de3e1c9da2ae2411a61235138f346331941530d7cae56b39f4a6497f7af03ed7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de3e1c9da2ae2411a61235138f346331941530d7cae56b39f4a6497f7af03ed7/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e169284b4a8e158327a8653f56ac3081327a00ad08954dab565ecc839277bed5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e169284b4a8e158327a8653f56ac3081327a00ad08954dab565ecc839277bed5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f481cf99c651eee61f335e49d463ec1d951374b2c5058f77e8c0c48cbe8a661f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f481cf99c651eee61f335e49d463ec1d951374b2c5058f77e8c0c48cbe8a661f/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22f84c42-8ee4-4133-a636-f02598d2b61b/volumes/kubernetes.io~projected/kube-api-access-vgw2f:{mountpoint:/var/lib/kubelet/pods/22f84c42-8ee4-4133-a636-f02598d2b61b/volumes/kubernetes.io~projected/kube-api-access-vgw2f major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~projected/kube-api-access-5gvfm:{mountpoint:/var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~projected/kube-api-access-5gvfm major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~projected/kube-api-access-rzt5g:{mountpoint:/var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~projected/kube-api-access-rzt5g major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed/volumes/kubernetes.io~projected/kube-api-access-dk29s:{mountpoint:/var/lib/kubelet/pods/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed/volumes/kubernetes.io~projected/kube-api-access-dk29s major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~projected/kube-api-access-pc7x7:{mountpoint:/var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~projected/kube-api-access-pc7x7 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~projected/kube-api-access-8cnqm:{mountpoint:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~projected/kube-api-access-8cnqm major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8/volumes/kubernetes.io~projected/kube-api-access-wsrhp:{mountpoint:/var/lib/kubelet/pods/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8/volumes/kubernetes.io~projected/kube-api-access-wsrhp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8/volumes/kubernetes.io~secret/metrics-certs major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74da9552-2470-4a9f-8160-d3a29aa5d9f5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/74da9552-2470-4a9f-8160-d3a29aa5d9f5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7648ae20-ac24-49f7-a8a7-11b204b9a2b3/volumes/kubernetes.io~projected/kube-api-access-vvddc:{mountpoint:/var/lib/kubelet/pods/7648ae20-ac24-49f7-a8a7-11b204b9a2b3/volumes/kubernetes.io~projected/kube-api-access-vvddc major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76c3a586-fa6f-42bd-a0d8-42211a9ae53a/volumes/kubernetes.io~projected/kube-api-access-c2ljx:{mountpoint:/var/lib/kubelet/pods/76c3a586-fa6f-42bd-a0d8-42211a9ae53a/volumes/kubernetes.io~projected/kube-api-access-c2ljx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76c3a586-fa6f-42bd-a0d8-42211a9ae53a/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/76c3a586-fa6f-42bd-a0d8-42211a9ae53a/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ebd138c-3a98-4268-b843-7b18f6ab68f8/volumes/kubernetes.io~projected/kube-api-access-9jzjj:{mountpoint:/var/lib/kubelet/pods/7ebd138c-3a98-4268-b843-7b18f6ab68f8/volumes/kubernetes.io~projected/kube-api-access-9jzjj major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ebd138c-3a98-4268-b843-7b18f6ab68f8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7ebd138c-3a98-4268-b843-7b18f6ab68f8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~projected/kube-api-access-vlgtj:{mountpoint:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~projected/kube-api-access-vlgtj major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b314a49f-8d9b-41f7-81f6-67f6c486f9b4/volumes/kubernetes.io~projected/kube-api-access-hlwps:{mountpoint:/var/lib/kubelet/pods/b314a49f-8d9b-41f7-81f6-67f6c486f9b4/volumes/kubernetes.io~projected/kube-api-access-hlwps major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9/volumes/kubernetes.io~projected/kube-api-access-ln994:{mountpoint:/var/lib/kubelet/pods/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9/volumes/kubernetes.io~projected/kube-api-access-ln994 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc3ac303-6d4c-4f0a-9279-2b3c428adccd/volumes/kubernetes.io~projected/kube-api-access-hcfv6:{mountpoint:/var/lib/kubelet/pods/cc3ac303-6d4c-4f0a-9279-2b3c428adccd/volumes/kubernetes.io~projected/kube-api-access-hcfv6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volumes/kubernetes.io~projected/kube-api-access-8ktm5:{mountpoint:/var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volumes/kubernetes.io~projected/kube-api-access-8ktm5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e453e98d-c621-4498-ad81-8cbc7eee10fe/volumes/kubernetes.io~projected/kube-api-access-kwd52:{mountpoint:/var/lib/kubelet/pods/e453e98d-c621-4498-ad81-8cbc7eee10fe/volumes/kubernetes.io~projected/kube-api-access-kwd52 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed940156-2f94-4372-a949-f07e265f0c8b/volumes/kubernetes.io~projected/kube-api-access-jph7w:{mountpoint:/var/lib/kubelet/pods/ed940156-2f94-4372-a949-f07e265f0c8b/volumes/kubernetes.io~projected/kube-api-access-jph7w major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efae3ac8-4b9e-4598-99b0-33eb10cd1935/volumes/kubernetes.io~projected/kube-api-access-sk6gk:{mountpoint:/var/lib/kubelet/pods/efae3ac8-4b9e-4598-99b0-33eb10cd1935/volumes/kubernetes.io~projected/kube-api-access-sk6gk major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efae3ac8-4b9e-4598-99b0-33eb10cd1935/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/efae3ac8-4b9e-4598-99b0-33eb10cd1935/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/20e8035b52a051ad8ea2535ccac0c267d0ff71e5fd5e26f1d7d602d77129534b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b238adb49a271633a9c956e46c91f3e99659c734a42ff81ca942ddac48f9cfeb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9c2549e26431b5eeafed9682cbb69aef715c0e5970b7dae08f612c5dc91488e3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1a54ec9eb290412f95e94480c5cf4994b617df3215a790dde6db6cdf3f6cb17c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2f0588feb7eef9b077f2f8d8c4f5d2bd87ff339612c91feca2726a7ec6b11364/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9d9ee75729c656c71c4894db7bbd4be92bb84d7d4a57d34d7e4f494537a0524c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c4e50bdae835ccc1d3cb743e189f38d47d3bda0aa58f1408f5a28fbf78f48a52/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/61be0e1ac69e8b496f4fa0373d9a99e373c631afdbf00de47dbd2878d1940a2c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/abc4ee5d8e2fe295eddb47d2d59318f6184b878e9dd3a36e0f83c4f7970ef2d2/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1c1375387fa162fcb75e0d614caaf3edb3511032d748fb29c708c4e8ddbbd8e9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/9f0a7278f2fd932bb0f04307de2e15074685f36c739f5da3a102749dc906ae3f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/375a12dc564936ce730283d8337f4ec9731a8c1c361e9db8f9fd74cf41dfb4ab/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b3606e261c97a6d6dad7e2abdede26dbd4b8ded79c90eb9a9dfb69455c7e8e19/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ac7b0ec8819bf00ce1f9eed2e7baea763bc0956e17dd40227127ab6bdb0ffeb2/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a69d5ee3a5199668da6a83679d514e81c645a5cc80b9ec82b6fab01e1fa37276/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2a96aa5491f8663f59f3eee0e48395461f2704774cada0f2427a8010a2fda4a6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/924649798c54e79a5054d7a0d9bda77d9014507bc2a9613f34f42c37db9bbfb8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/34369d8bfb955ee9083356f379be48b335491d4c81c849f8b336a74ee5589d07/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/100290e7550aba1bf9f04d3d003c9375319bb7a5aeb1a633a980b8cc41988003/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e78e7e05f933c7f8e49a1312884023ff7e9ae44d20c38845617c494b0a7bdabf/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/3b167210a64b185a3f24ce68b28e0d8d11c7b6a3b32e348bcabf1349a7de1511/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4d53c07f5713e1c70234d94ea9f61ffe85378c648a60b5a4682d1937501b81a4/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/1cc142e5c3ab243279ddbb07dfddb4a846401a9afdf231a9622fe337c502ba30/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/4cf3ac3be0c851f1cd439a69a548597b3f23993b3b30d6bc550c856f03f7e84b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/e792694642b4b6f10426bb192f2d64078eff66ff9f68aff6548f602e5dd480c0/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/ca17fc507e0cd2328a061f5f3f95fa36407a85b579f3e70bdb7bfa4976647535/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/25a0a577b84ea5defa32a00f677417d28789572a2c21f06726697bd307d2bbc5/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/6d7fa20bcc25da0f8f8e49f5bf3518aac3d97348edfc0dcfb9c0dfc427da9ad1/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/a814612a44b0e485f88d12338daf77c4b1fd18da1cb0a257e12c1c931863da1e/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/b219625a1b66ce8f77c3f56c3c46edf0adbe61ac74c3a70c89bc841763dd91cb/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/b9db1f98d1e88ac1a12ec4a5032c6a826afeeb267600c7277812715ce468dfb2/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/7744bf94e2c8e63369e0a682ced593e67321076f56064c01aa4de7a3cfcb37a5/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/c1541c7c35b8f71cf98d9ac5b3d4611b1b2a4ca3246b999ca3ad76d29d00f5db/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/ded0d0a90d015b6e0c622a0967ce4ab5d05253477bbb7b0ac0ef5a366a254dfc/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/65137be61641e4b7529db8f00c0d2ab2b4f3f80eaa365b987ee05ae3b1517b73/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/c2ca42fd1f470c7bdb793a66b15e7f42ae4efd80d2f9822ee5b00907c96d901e/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/48335ba090dfa4506fcc207bfc3717d24b158875ca4c396abc2c5b7fe499607f/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/0b4c26c9e8b88d58b0af45712d65d2b83fbfdc1ecf86e081eb68437a01823b32/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/fb9938aac90125cbecf4e3f4420e9a9ace96971907736f6bddfa179b4856e5e9/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/62956a6574ab78bd74e6bee17f3912e2684986ca3536f36cf74c46f7205b4438/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/16b03f9faeb070cb6e31c7582dc9872118b009f7d1ec44f7c10624a69a524f42/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/10716b8bad4904d235339ef29ff86704f8e9c7b833ae04b42bf64b7f3111f33b/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/0474052c02a655ed6fc7eb1a906da3a4a63df2e1e2e7fd01e0a1ba0ec8019cf7/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/0b7508027b956176a5253a0709d65a6d6f6702a3c5e7777693dba4431edd5338/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/1c376ff05e8165f18ab82bddb4024f4db1f59d5971661c928f1d203b2f68aaf4/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2832c16e2a52af60658f6ae7fa543d02b893f2bd113a5762440c0792c361cb69/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a83a7af04a179f04812e24d6f0f303f48842b38d4cd05f89b2a8b10b06c42283/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0690a778cacd974f1b189406073566fdd44d4485ddf2f5be0e31832eb2ce4099/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c68cb1eabfc1b3f85a6e0ec7cc9a5f5f02224c78ceb32266866a6ac68e337c4c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d0c39feddd12d0de0fcbb4bad8518ad656d82a88866fe12d7d9c9494b95392c3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9e2c67ba1e8451e3ec91fa3e4487ecd429de64d1e25e3b8fc0534f775eb5dfd3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/40feee522ffc04d96de58553eac92e3d0872a2f60fcc341088320668043cb20d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/27527b104d0215063803e16c31673838c842b750f6811858bd7c0eb72630b4ab/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cbcf4cf77cbecb87d47ab2a907fb831f18523492a735505ef8dba40688c2775c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5cce6b824ab9ddde7b7a3aad2a68ef443a035c5feade4f13beee56cd14af3da5/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 22:06:03.815212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.814765 5109 manager.go:217] Machine: {Timestamp:2026-11-02 22:06:03.814044771 +0000 UTC m=+0.079037981 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100307 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254c6bc94ee193eb0c96c09840c9f5 SystemUUID:ec254c6b-c94e-e193-eb0c-96c09840c9f5 BootID:7a064eec-d7cf-4bc1-8456-ee1972e997fc Filesystems:[{Device:/run/containers/storage/overlay-containers/89e7a1e5f423c963456d66b6293e7c2c5220df7af479ca412e548dc1205b2fcd/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a8b00d56db228da808fa1c26314ce2d562f10d6a197c620fc47845e90d6e881/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e169284b4a8e158327a8653f56ac3081327a00ad08954dab565ecc839277bed5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efae3ac8-4b9e-4598-99b0-33eb10cd1935/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b314a49f-8d9b-41f7-81f6-67f6c486f9b4/volumes/kubernetes.io~projected/kube-api-access-hlwps DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~projected/kube-api-access-pc7x7 DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/709bb7207c08474bd8a2b604c828381882bafe8fa2c37dd8dc81149905f36435/userdata/shm DeviceMajor:0 DeviceMinor:275 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:306 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~projected/kube-api-access-rzt5g DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c8a543704dd23caa16fecdecdc566ec9410669a2f09c6196ed5a0da8621d9d0/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76c3a586-fa6f-42bd-a0d8-42211a9ae53a/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de3e1c9da2ae2411a61235138f346331941530d7cae56b39f4a6497f7af03ed7/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/ed940156-2f94-4372-a949-f07e265f0c8b/volumes/kubernetes.io~projected/kube-api-access-jph7w DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/051c1af1375591ce8b9003110ee894ae74962a0d86f33ded706a979592c68342/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~projected/kube-api-access-5gvfm DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc3ac303-6d4c-4f0a-9279-2b3c428adccd/volumes/kubernetes.io~projected/kube-api-access-hcfv6 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60d62e41334654822970b4cda017d2643cf56db8869f668311d8826c21bbabd6/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5303ac1ad52ac0bfe694e1759e2af566d7b4d5bb4d877ece111dd9c71b9ee98/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/034147492ccae2628bf3132c590351fbe991e7a1563f5cb23b5d2f474160b86e/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cda3330bd7923fa858ca326092dc5919a2888505c1da96d4d66dbe4677ab8247/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54880730-d3aa-4d76-8234-0dc766363113/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0d9566f1ba5727be81ce53a4d678b4e57d6593a37f89305f736482e2ae42a5d/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/602b835d9710f35a787b961fe64a02e27a92d524182584a8594ad05eee562bb1/userdata/shm DeviceMajor:0 DeviceMinor:308 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:303 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed/volumes/kubernetes.io~projected/kube-api-access-dk29s DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62e7c81fbbf8353a15294b18136e861f5526541c49215151926abc72e867d3b7/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ebd138c-3a98-4268-b843-7b18f6ab68f8/volumes/kubernetes.io~projected/kube-api-access-9jzjj DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/76c3a586-fa6f-42bd-a0d8-42211a9ae53a/volumes/kubernetes.io~projected/kube-api-access-c2ljx DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:304 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/306b1470-ddb8-4e0c-b450-3c2ddb541faf/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7648ae20-ac24-49f7-a8a7-11b204b9a2b3/volumes/kubernetes.io~projected/kube-api-access-vvddc DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8/volumes/kubernetes.io~projected/kube-api-access-wsrhp DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:298 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd19933d1a573d9cdaec74f596f7fa7384455d80c4b43f431c28ef0bee62ac0e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c686753802cdb04ce0dc0a473dc992383809915f411027a2f480c484adab164/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:305 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ebd138c-3a98-4268-b843-7b18f6ab68f8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~projected/kube-api-access-8cnqm DeviceMajor:0 DeviceMinor:307 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e2290f26a1c7b0a1235abbf0965ef1f5bb30cf0a16f024547401fb76a29890b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~projected/kube-api-access-vlgtj DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63b6f934-07bb-42c1-8e3d-5bc2369b1716/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:302 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74da9552-2470-4a9f-8160-d3a29aa5d9f5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:265 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22f84c42-8ee4-4133-a636-f02598d2b61b/volumes/kubernetes.io~projected/kube-api-access-vgw2f DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f481cf99c651eee61f335e49d463ec1d951374b2c5058f77e8c0c48cbe8a661f/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/28191529-9de8-4283-bd79-6b4de730919e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cc496e-e59e-4f60-87bd-eeb2b18a165c/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e453e98d-c621-4498-ad81-8cbc7eee10fe/volumes/kubernetes.io~projected/kube-api-access-kwd52 DeviceMajor:0 DeviceMinor:270 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/efae3ac8-4b9e-4598-99b0-33eb10cd1935/volumes/kubernetes.io~projected/kube-api-access-sk6gk DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ed1d10-fcee-4a00-a544-efb9346dac66/volumes/kubernetes.io~projected/kube-api-access-8ktm5 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9/volumes/kubernetes.io~projected/kube-api-access-ln994 DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:269 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:034147492ccae26 MacAddress:5e:a7:b9:2b:7c:c1 Speed:10000 Mtu:8901} {Name:602b835d9710f35 MacAddress:86:4e:14:99:a8:9c Speed:10000 Mtu:8901} {Name:62e7c81fbbf8353 MacAddress:2e:4f:8d:fa:a4:5f Speed:10000 Mtu:8901} {Name:709bb7207c08474 MacAddress:36:62:d4:60:0d:0b Speed:10000 Mtu:8901} {Name:8a8b00d56db228d MacAddress:06:5c:d7:9e:ea:f4 Speed:10000 Mtu:8901} {Name:9c686753802cdb0 MacAddress:9a:bd:c3:60:06:91 Speed:10000 Mtu:8901} {Name:9c8a543704dd23c MacAddress:3e:1f:9f:87:5e:db Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e5:24:01:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:02:96:40:e3:a3:56 Speed:0 Mtu:8901} {Name:cda3330bd7923fa MacAddress:ea:01:04:1c:e5:dc Speed:10000 Mtu:8901} {Name:de3e1c9da2ae241 MacAddress:32:49:5c:15:41:53 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e5:24:01:7d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:aa:2f:71:d7:ee:45 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:5b:6a:55:85:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:06:03.815212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815216 5109 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:06:03.815521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815286 5109 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:06:03.815711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815676 5109 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:06:03.815940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815714 5109 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-75.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:06:03.815940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815942 5109 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:06:03.816142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815955 5109 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:06:03.816142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815970 5109 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:06:03.816142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.815999 5109 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:06:03.816275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816154 5109 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:06:03.816419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816404 5109 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:06:03.816555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816543 5109 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:06:03.816657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816622 5109 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:06:03.816657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816637 5109 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:06:03.816657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816656 5109 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:06:03.816793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816671 5109 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:06:03.816793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.816685 5109 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:06:03.818223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.818201 5109 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:06:03.818320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.818227 5109 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:06:03.819937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.819243 5109 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:06:03.834856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.834826 5109 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:06:03.834856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.834849 5109 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:06:03.835054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.834989 5109 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 22:06:03.835292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835268 5109 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:06:03.835414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835307 5109 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:06:03.835482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835449 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:06:03.835482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835468 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:06:03.835482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835478 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835488 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835503 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835512 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835522 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835533 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835544 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835554 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835594 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835614 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835656 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:06:03.835660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.835665 5109 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:06:03.840599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.840275 5109 server.go:1263] "Started kubelet" Nov 02 22:06:03.840599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.840373 5109 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:06:03.840763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.840511 5109 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:06:03.840763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.840683 5109 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:06:03.841333 ip-10-0-143-75 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:06:03.841457 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.841426 5109 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:06:03.842102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.841707 5109 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:06:03.844120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.844103 5109 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:06:03.844722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.844698 5109 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:06:03.844722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.844721 5109 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:06:03.844861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.844687 5109 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:06:03.844861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.844800 5109 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:06:03.844989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.844935 5109 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:59:52 +0000 UTC" deadline="2028-04-20 07:54:59.590252857 +0000 UTC" Nov 02 22:06:03.845045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.844989 5109 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12825h48m55.745268191s" Nov 02 22:06:03.845117 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.845087 5109 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849099 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ebd138c-3a98-4268-b843-7b18f6ab68f8" volumeName="kubernetes.io/empty-dir/7ebd138c-3a98-4268-b843-7b18f6ab68f8-data-volume" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849144 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9" volumeName="kubernetes.io/secret/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-cert" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849154 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc3ac303-6d4c-4f0a-9279-2b3c428adccd" volumeName="kubernetes.io/configmap/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-serviceca" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849162 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc3ac303-6d4c-4f0a-9279-2b3c428adccd" volumeName="kubernetes.io/projected/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-kube-api-access-hcfv6" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849169 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e453e98d-c621-4498-ad81-8cbc7eee10fe" volumeName="kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849177 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28191529-9de8-4283-bd79-6b4de730919e" volumeName="kubernetes.io/configmap/28191529-9de8-4283-bd79-6b4de730919e-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849185 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28191529-9de8-4283-bd79-6b4de730919e" volumeName="kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-tls" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849198 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed" volumeName="kubernetes.io/configmap/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-iptables-alerter-script" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849208 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/projected/63b6f934-07bb-42c1-8e3d-5bc2369b1716-kube-api-access-8cnqm" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849215 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849223 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76c3a586-fa6f-42bd-a0d8-42211a9ae53a" volumeName="kubernetes.io/configmap/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-config-volume" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849231 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54880730-d3aa-4d76-8234-0dc766363113" volumeName="kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-tls" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849239 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849246 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74da9552-2470-4a9f-8160-d3a29aa5d9f5" volumeName="kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849260 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7648ae20-ac24-49f7-a8a7-11b204b9a2b3" volumeName="kubernetes.io/projected/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kube-api-access-vvddc" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849268 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed940156-2f94-4372-a949-f07e265f0c8b" volumeName="kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-binary-copy" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849275 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f84c42-8ee4-4133-a636-f02598d2b61b" volumeName="kubernetes.io/empty-dir/22f84c42-8ee4-4133-a636-f02598d2b61b-tmp-dir" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849283 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54880730-d3aa-4d76-8234-0dc766363113" volumeName="kubernetes.io/projected/54880730-d3aa-4d76-8234-0dc766363113-kube-api-access-pc7x7" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849290 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849297 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849304 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849311 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b314a49f-8d9b-41f7-81f6-67f6c486f9b4" volumeName="kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-daemon-config" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849318 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efae3ac8-4b9e-4598-99b0-33eb10cd1935" volumeName="kubernetes.io/secret/efae3ac8-4b9e-4598-99b0-33eb10cd1935-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849330 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54880730-d3aa-4d76-8234-0dc766363113" volumeName="kubernetes.io/empty-dir/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-textfile" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849337 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-grpc-tls" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849344 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" volumeName="kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849354 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76c3a586-fa6f-42bd-a0d8-42211a9ae53a" volumeName="kubernetes.io/secret/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-metrics-tls" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849360 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed940156-2f94-4372-a949-f07e265f0c8b" volumeName="kubernetes.io/projected/ed940156-2f94-4372-a949-f07e265f0c8b-kube-api-access-jph7w" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849371 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/configmap/63b6f934-07bb-42c1-8e3d-5bc2369b1716-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849379 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ed1d10-fcee-4a00-a544-efb9346dac66" volumeName="kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-script-lib" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849387 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306b1470-ddb8-4e0c-b450-3c2ddb541faf" volumeName="kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-tmp" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849395 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ebd138c-3a98-4268-b843-7b18f6ab68f8" volumeName="kubernetes.io/secret/7ebd138c-3a98-4268-b843-7b18f6ab68f8-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849402 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849409 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ed1d10-fcee-4a00-a544-efb9346dac66" volumeName="kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-env-overrides" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849416 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54880730-d3aa-4d76-8234-0dc766363113" volumeName="kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849423 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/empty-dir/63b6f934-07bb-42c1-8e3d-5bc2369b1716-tmp" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849431 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849438 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8" volumeName="kubernetes.io/projected/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-kube-api-access-wsrhp" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849444 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ebd138c-3a98-4268-b843-7b18f6ab68f8" volumeName="kubernetes.io/configmap/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849452 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ebd138c-3a98-4268-b843-7b18f6ab68f8" volumeName="kubernetes.io/projected/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-api-access-9jzjj" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849459 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849466 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a" volumeName="kubernetes.io/secret/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-agent-certs" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849475 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54880730-d3aa-4d76-8234-0dc766363113" volumeName="kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849482 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ed1d10-fcee-4a00-a544-efb9346dac66" volumeName="kubernetes.io/projected/d6ed1d10-fcee-4a00-a544-efb9346dac66-kube-api-access-8ktm5" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849489 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306b1470-ddb8-4e0c-b450-3c2ddb541faf" volumeName="kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-tuned" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849497 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306b1470-ddb8-4e0c-b450-3c2ddb541faf" volumeName="kubernetes.io/projected/306b1470-ddb8-4e0c-b450-3c2ddb541faf-kube-api-access-rzt5g" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849504 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54880730-d3aa-4d76-8234-0dc766363113" volumeName="kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849511 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-federate-client-tls" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849518 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9" volumeName="kubernetes.io/projected/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-kube-api-access-ln994" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849524 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ed1d10-fcee-4a00-a544-efb9346dac66" volumeName="kubernetes.io/secret/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849533 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ed1d10-fcee-4a00-a544-efb9346dac66" volumeName="kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-config" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849540 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f84c42-8ee4-4133-a636-f02598d2b61b" volumeName="kubernetes.io/projected/22f84c42-8ee4-4133-a636-f02598d2b61b-kube-api-access-vgw2f" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849547 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/projected/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-kube-api-access-vlgtj" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849554 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed940156-2f94-4372-a949-f07e265f0c8b" volumeName="kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849561 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efae3ac8-4b9e-4598-99b0-33eb10cd1935" volumeName="kubernetes.io/projected/efae3ac8-4b9e-4598-99b0-33eb10cd1935-kube-api-access-sk6gk" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849594 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28191529-9de8-4283-bd79-6b4de730919e" volumeName="kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849611 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76c3a586-fa6f-42bd-a0d8-42211a9ae53a" volumeName="kubernetes.io/empty-dir/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-tmp-dir" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849619 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b314a49f-8d9b-41f7-81f6-67f6c486f9b4" volumeName="kubernetes.io/projected/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-kube-api-access-hlwps" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849627 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed940156-2f94-4372-a949-f07e265f0c8b" volumeName="kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849641 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed" volumeName="kubernetes.io/projected/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-kube-api-access-dk29s" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849650 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 22:06:03.849603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849657 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b6f934-07bb-42c1-8e3d-5bc2369b1716" volumeName="kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 22:06:03.851726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849664 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b314a49f-8d9b-41f7-81f6-67f6c486f9b4" volumeName="kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cni-binary-copy" seLinuxMountContext="" Nov 02 22:06:03.851726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849671 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76c3a586-fa6f-42bd-a0d8-42211a9ae53a" volumeName="kubernetes.io/projected/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-kube-api-access-c2ljx" seLinuxMountContext="" Nov 02 22:06:03.851726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849678 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cc496e-e59e-4f60-87bd-eeb2b18a165c" volumeName="kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-client-tls" seLinuxMountContext="" Nov 02 22:06:03.851726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849690 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a" volumeName="kubernetes.io/configmap/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-konnectivity-ca" seLinuxMountContext="" Nov 02 22:06:03.851726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849699 5109 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28191529-9de8-4283-bd79-6b4de730919e" volumeName="kubernetes.io/projected/28191529-9de8-4283-bd79-6b4de730919e-kube-api-access-5gvfm" seLinuxMountContext="" Nov 02 22:06:03.851726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849709 5109 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:06:03.851726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.849717 5109 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:06:03.853731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.853711 5109 factory.go:55] Registering systemd factory Nov 02 22:06:03.853874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.853734 5109 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:06:03.854043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.854030 5109 factory.go:153] Registering CRI-O factory Nov 02 22:06:03.854117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.854047 5109 factory.go:223] Registration of the crio container factory successfully Nov 02 22:06:03.854117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.854101 5109 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:06:03.854226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.854121 5109 factory.go:103] Registering Raw factory Nov 02 22:06:03.854226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.854138 5109 manager.go:1199] Started watching for new ooms in manager Nov 02 22:06:03.854443 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.854430 5109 manager.go:319] Starting recovery of all containers Nov 02 22:06:03.883669 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.880642 5109 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:06:03.883980 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.883953 5109 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:06:03.883980 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.883980 5109 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:06:03.884104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.884002 5109 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:06:03.884104 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.884044 5109 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 22:06:03.891528 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.891494 5109 generic.go:356] "Generic (PLEG): container finished" podID="54880730-d3aa-4d76-8234-0dc766363113" containerID="c6dbecbc591073c45860f43d95775f323423dd4cb7e3339348364657aabfccb3" exitCode=0 Nov 02 22:06:03.899631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.899587 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="258c146ecb32a4d929348378f40c15006c61311707cf2442206831e3b383adb5" exitCode=0 Nov 02 22:06:03.899631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.899628 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="d857fa2ca218e11ac36e8cc7447e5ce64d018889adc59564c059caf8fa9cff12" exitCode=0 Nov 02 22:06:03.899631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.899638 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="892efecf0729e302bf14be10c1ad95462ca2d2449f538af519343a3d9fbea367" exitCode=0 Nov 02 22:06:03.899892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.899648 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="9b20aaeee09c1817bbf1c994439e58b7c4ae1db1adb24ecde1cc649b12f08cb2" exitCode=0 Nov 02 22:06:03.899892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.899659 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="52c91ca65299b795dd70711236d23d755ad7844c27986db7d7663ef8b1c70343" exitCode=0 Nov 02 22:06:03.899892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.899665 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed940156-2f94-4372-a949-f07e265f0c8b" containerID="afdf8f5dd2874da460d1e31cb7e7ed1dc698701392de78ba5914f96a42e4d5fd" exitCode=0 Nov 02 22:06:03.922923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.922890 5109 generic.go:356] "Generic (PLEG): container finished" podID="87c23d8b6ee5b61bb2ffa790b47abb66" containerID="fc93a35565f4a587049cca83ef1faa9cb191a7b3564712f26ec479f1fd0a2beb" exitCode=0 Nov 02 22:06:03.944151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.944123 5109 manager.go:324] Recovery completed Nov 02 22:06:03.945201 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.945177 5109 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:06:03.962690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.962661 5109 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:03.965473 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.965451 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:03.965607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.965485 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:03.965607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.965496 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:03.967690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967670 5109 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:06:03.967690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967686 5109 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:06:03.967690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967686 5109 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:03.967875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967705 5109 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:06:03.967909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967878 5109 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 22:06:03.967937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967897 5109 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 22:06:03.967937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967919 5109 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 22:06:03.967937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967924 5109 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 22:06:03.967937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967929 5109 policy_none.go:50] "Start" Nov 02 22:06:03.967937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967938 5109 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:06:03.968089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.967949 5109 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:06:03.968836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.968818 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:03.968920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.968854 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:03.968920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.968869 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:03.969206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.969191 5109 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 22:06:03.969206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.969207 5109 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 22:06:03.969286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.969213 5109 policy_none.go:44] "Start" Nov 02 22:06:03.973841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.973814 5109 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:06:03.976476 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.976460 5109 manager.go:341] "Starting Device Plugin manager" Nov 02 22:06:03.976546 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.976489 5109 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:06:03.976546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.976498 5109 server.go:87] "Starting device plugin registration server" Nov 02 22:06:03.976775 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.976763 5109 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:06:03.976806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.976779 5109 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:06:03.976905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.976876 5109 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:06:03.977083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.977042 5109 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:06:03.977148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.977097 5109 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:06:03.977148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.977121 5109 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:06:03.978021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.977181 5109 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:06:03.977081309Z","UUID":"d5c554dc-f9ea-4778-b8eb-d3605cf44e39","Handler":null,"Name":"","Endpoint":""} Nov 02 22:06:03.978432 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.978404 5109 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:06:03.978519 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.978447 5109 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" Nov 02 22:06:03.979389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.979371 5109 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:06:03.979471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.979398 5109 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:06:03.984129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.984097 5109 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal"] Nov 02 22:06:03.984244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.984232 5109 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:03.985234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.985215 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:03.985327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.985252 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:03.985327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.985270 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:03.985587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.985551 5109 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:03.985702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.985689 5109 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:03.986621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.986605 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:03.986714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.986637 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:03.986714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.986651 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:03.986714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.986653 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:03.986714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.986680 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:03.986714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.986697 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:03.986936 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.986823 5109 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:03.987117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.987078 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"87c23d8b6ee5b61bb2ffa790b47abb66","Type":"ContainerStarted","Data":"84a78d3391b35b5c1ba1abf01bebcffa3eb51c89fd756e9c08470cfdac0d7386"} Nov 02 22:06:03.987181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.987116 5109 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:03.987181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.987134 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"87c23d8b6ee5b61bb2ffa790b47abb66","Type":"ContainerDied","Data":"fc93a35565f4a587049cca83ef1faa9cb191a7b3564712f26ec479f1fd0a2beb"} Nov 02 22:06:03.987181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.987158 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"87c23d8b6ee5b61bb2ffa790b47abb66","Type":"ContainerStarted","Data":"6384396bea74590d3990e8556ad185adf177cc382e00c938801c98c2ed49ddb7"} Nov 02 22:06:03.987323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.987188 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"52559cd016dd3e2df41639aa852bc053","Type":"ContainerStarted","Data":"9a2bad957f7798870503b821c438f36b8b74e1502ad4d2245f134afd6e235205"} Nov 02 22:06:03.987323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.987206 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" event={"ID":"52559cd016dd3e2df41639aa852bc053","Type":"ContainerStarted","Data":"7bb51ca92dd099b66afeaa2b439d94c68ca63f4ab20fb57eafbda723c51a7773"} Nov 02 22:06:03.988215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.988199 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:03.988298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.988231 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:03.988298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:03.988246 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:03.988411 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:03.988395 5109 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-75.us-west-2.compute.internal\" not found" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.020920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.020881 5109 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:04.021071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.020936 5109 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:04.021105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.021079 5109 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 22:06:04.021656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.021639 5109 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:04.045106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.045068 5109 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.077770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.077733 5109 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:04.080041 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.080021 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:04.080106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.080056 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:04.080106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.080066 5109 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-75.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:04.080180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.080146 5109 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.113499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.113469 5109 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:04.114114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.114093 5109 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:04.114194 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:04.114142 5109 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.114194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.114156 5109 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.121201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.121181 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52559cd016dd3e2df41639aa852bc053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"52559cd016dd3e2df41639aa852bc053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.121249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.121209 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.121249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.121228 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.184060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.183968 5109 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.184060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.184061 5109 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.203682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.203646 5109 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:04.203868 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:04.203722 5109 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.221734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.221692 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.221734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.221736 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.221964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.221764 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52559cd016dd3e2df41639aa852bc053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"52559cd016dd3e2df41639aa852bc053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.221964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.221812 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52559cd016dd3e2df41639aa852bc053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"52559cd016dd3e2df41639aa852bc053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.221964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.221818 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.221964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.221830 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c23d8b6ee5b61bb2ffa790b47abb66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" (UID: \"87c23d8b6ee5b61bb2ffa790b47abb66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.818515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.818479 5109 apiserver.go:52] "Watching apiserver" Nov 02 22:06:04.936650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.936624 5109 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.936789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.936736 5109 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.964264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.964232 5109 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:06:04.964839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.964816 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7sslx","openshift-multus/multus-additional-cni-plugins-55xk7","openshift-dns/dns-default-vxpxh","openshift-dns/node-resolver-6lwq6","openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw","openshift-network-diagnostics/network-check-target-pb5sz","openshift-network-operator/iptables-alerter-47zpv","kube-system/global-pull-secret-syncer-knp2v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp","openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq","openshift-ingress-canary/ingress-canary-8zm4n","openshift-insights/insights-runtime-extractor-p27gn","openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc","openshift-monitoring/telemeter-client-674c5d5fcb-845r7","openshift-ovn-kubernetes/ovnkube-node-5xqr8","kube-system/konnectivity-agent-zzcqw","openshift-cluster-node-tuning-operator/tuned-n9jzr","openshift-image-registry/node-ca-lsbsf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal","openshift-multus/multus-xf2bk","openshift-multus/network-metrics-daemon-h7lsc","openshift-console/console-584dc6f89b-wtdxr","kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal"] Nov 02 22:06:04.976696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.976665 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:04.979846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.979825 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:04.997870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.997835 5109 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:04.998015 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:04.997904 5109 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:04.998454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:04.998436 5109 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:04.998557 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:04.998495 5109 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.us-west-2.compute.internal" Nov 02 22:06:05.014668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.014641 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:06:05.015997 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.015975 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:06:05.015997 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.016000 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrhfb\"" Nov 02 22:06:05.016353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.016336 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:06:05.017473 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.017454 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:06:05.047394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.047365 5109 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:06:05.101462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101357 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:06:05.101462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101357 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:06:05.101791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101501 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 22:06:05.101791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101618 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 22:06:05.101791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101732 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcntw\"" Nov 02 22:06:05.101986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101874 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-klqgw\"" Nov 02 22:06:05.101986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101906 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4rxp8\"" Nov 02 22:06:05.101986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101907 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 22:06:05.101986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101923 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 22:06:05.101986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.101951 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6jxx8\"" Nov 02 22:06:05.102386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102039 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:06:05.102386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102246 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 22:06:05.102386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102263 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8d44q\"" Nov 02 22:06:05.102386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102302 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4gtmj\"" Nov 02 22:06:05.102386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102329 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 22:06:05.102386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102333 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 22:06:05.102386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102359 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5sg6\"" Nov 02 22:06:05.102867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102421 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 22:06:05.102867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102447 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 22:06:05.102867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102591 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4ts5p\"" Nov 02 22:06:05.102867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102742 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 22:06:05.102867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102772 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wsgcd\"" Nov 02 22:06:05.102867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102797 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4knj\"" Nov 02 22:06:05.102867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102841 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g764w\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102897 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rksbh\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102940 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ddaq9n2d3jel1\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.102989 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-656w2\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103083 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103097 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rsfpb\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103107 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103109 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103151 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fcg7g\"" Nov 02 22:06:05.103238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103186 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103287 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nh6bm\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103297 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103327 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103357 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103399 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103417 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103416 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103437 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t4sw7\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103485 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zsfxd\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103513 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5dnx\"" Nov 02 22:06:05.103600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.103591 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9shnq\"" Nov 02 22:06:05.115632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.115611 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:06:05.115766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.115735 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.115871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.115825 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.115871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.115846 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.115871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.115846 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 22:06:05.115871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.115846 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.116253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116232 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.116417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116270 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 22:06:05.116417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116337 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.116417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116338 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.116417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116355 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:06:05.116634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116423 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.116634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116444 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.116634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.116545 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 22:06:05.127089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127063 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-host\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:06:05.127168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127097 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.127168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127119 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.127240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127177 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-wtmp\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.127270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127237 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrhp\" (UniqueName: \"kubernetes.io/projected/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-kube-api-access-wsrhp\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:06:05.127334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127315 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-serving-certs-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.127424 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127408 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.127462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127427 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jph7w\" (UniqueName: \"kubernetes.io/projected/ed940156-2f94-4372-a949-f07e265f0c8b-kube-api-access-jph7w\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.127579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127481 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-lib-modules\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.127579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127545 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzt5g\" (UniqueName: \"kubernetes.io/projected/306b1470-ddb8-4e0c-b450-3c2ddb541faf-kube-api-access-rzt5g\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.127658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127619 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-serving-certs-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.127658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127624 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-serviceca\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:06:05.127723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127707 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-config-volume\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.127791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127770 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvddc\" (UniqueName: \"kubernetes.io/projected/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kube-api-access-vvddc\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.127824 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127807 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.127864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127841 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk29s\" (UniqueName: \"kubernetes.io/projected/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-kube-api-access-dk29s\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:06:05.127897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127863 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-oauth-config\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.127897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127866 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-config-volume\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.127897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127878 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-k8s-cni-cncf-io\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.127978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.127923 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-cert\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:06:05.128134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128114 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:06:05.128173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128117 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-secret-telemeter-client\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.128173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128153 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-run\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.128233 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128191 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-var-lib-kubelet\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.128233 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128216 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.128287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128238 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cnibin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.128287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128269 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-cert\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:06:05.128287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128275 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-serving-cert\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.128420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128337 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.128420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128381 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6gk\" (UniqueName: \"kubernetes.io/projected/efae3ac8-4b9e-4598-99b0-33eb10cd1935-kube-api-access-sk6gk\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:05.128516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128418 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-slash\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.128516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128476 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-textfile\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.128516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128506 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-metrics-client-ca\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.128688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128548 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efae3ac8-4b9e-4598-99b0-33eb10cd1935-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:05.128688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128556 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-textfile\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.128688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128628 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-ovn\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.128836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128701 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktm5\" (UniqueName: \"kubernetes.io/projected/d6ed1d10-fcee-4a00-a544-efb9346dac66-kube-api-access-8ktm5\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.128836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128742 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b6f934-07bb-42c1-8e3d-5bc2369b1716-metrics-client-ca\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.128836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128781 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-trusted-ca-bundle\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.128836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128810 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ljx\" (UniqueName: \"kubernetes.io/projected/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-kube-api-access-c2ljx\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.128836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128751 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efae3ac8-4b9e-4598-99b0-33eb10cd1935-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:05.129070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128837 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f84c42-8ee4-4133-a636-f02598d2b61b-hosts-file\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:06:05.129070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128869 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.129070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128909 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-tmp\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.129070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.128982 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-systemd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.129070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129012 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgtj\" (UniqueName: \"kubernetes.io/projected/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-kube-api-access-vlgtj\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129087 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129096 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129107 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129126 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129141 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-conf-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129160 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln994\" (UniqueName: \"kubernetes.io/projected/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-kube-api-access-ln994\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129216 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvfm\" (UniqueName: \"kubernetes.io/projected/28191529-9de8-4283-bd79-6b4de730919e-kube-api-access-5gvfm\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129248 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129264 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-device-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129292 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-modprobe-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.129307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129301 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129322 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-dbus\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129377 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-metrics-certs\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129386 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-env-overrides\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129413 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-config\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129515 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ebd138c-3a98-4268-b843-7b18f6ab68f8-data-volume\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129531 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-env-overrides\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129540 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-netns\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129595 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129616 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ebd138c-3a98-4268-b843-7b18f6ab68f8-data-volume\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129624 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-script-lib\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129656 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-daemon-config\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129695 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f84c42-8ee4-4133-a636-f02598d2b61b-tmp-dir\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:06:05.129747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129713 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129773 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f84c42-8ee4-4133-a636-f02598d2b61b-tmp-dir\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129833 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129827 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-script-lib\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129871 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzjj\" (UniqueName: \"kubernetes.io/projected/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-api-access-9jzjj\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129888 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-daemon-config\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.129918 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-tmp-dir\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130016 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-bin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.130078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130061 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-tmp-dir\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.130302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130094 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-hostroot\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.130302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130128 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwps\" (UniqueName: \"kubernetes.io/projected/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-kube-api-access-hlwps\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.130302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130149 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28191529-9de8-4283-bd79-6b4de730919e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.130302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130189 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-kubernetes\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.130302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130215 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfv6\" (UniqueName: \"kubernetes.io/projected/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-kube-api-access-hcfv6\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:06:05.130302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130242 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.130302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130290 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ebd138c-3a98-4268-b843-7b18f6ab68f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.130549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130325 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-kubelet\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.130549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130371 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-systemd-units\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.130549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130396 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-log-socket\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.130549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130421 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-cnibin\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.130549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130463 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.130549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130467 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-metrics-tls\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.130764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130528 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-netd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.130764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130595 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ebd138c-3a98-4268-b843-7b18f6ab68f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.130764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130657 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-konnectivity-ca\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:06:05.130764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130694 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-grpc-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.130764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130728 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysconfig\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.130764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130755 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ebd138c-3a98-4268-b843-7b18f6ab68f8-crio-socket\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130776 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-metrics-tls\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130781 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130821 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-bin\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130861 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-federate-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130896 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130929 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-netns\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130959 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.131004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130983 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-root\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.130987 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-grpc-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131011 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-host-slash\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131037 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131097 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-federate-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131123 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-systemd\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131139 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-sys\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131186 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-client-tls\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131196 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovn-node-metrics-cert\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131243 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131316 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-tuned\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.131374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131336 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131402 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-tuned\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131417 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovn-node-metrics-cert\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131430 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-conf\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131463 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-tmp\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131504 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-etc-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131513 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-tls\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131526 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28191529-9de8-4283-bd79-6b4de730919e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131534 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-config\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131542 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306b1470-ddb8-4e0c-b450-3c2ddb541faf-tmp\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131562 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-system-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131630 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-iptables-alerter-script\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131673 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6ed1d10-fcee-4a00-a544-efb9346dac66-ovnkube-config\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131689 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131707 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131743 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131759 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-var-lib-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131784 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cni-binary-copy\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131802 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.131816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131819 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-system-cni-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131841 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131872 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131890 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131922 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131958 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-multus-certs\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131974 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-etc-kubernetes\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.131993 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgw2f\" (UniqueName: \"kubernetes.io/projected/22f84c42-8ee4-4133-a636-f02598d2b61b-kube-api-access-vgw2f\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132009 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7x7\" (UniqueName: \"kubernetes.io/projected/54880730-d3aa-4d76-8234-0dc766363113-kube-api-access-pc7x7\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132040 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b6f934-07bb-42c1-8e3d-5bc2369b1716-tmp\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132081 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74da9552-2470-4a9f-8160-d3a29aa5d9f5-original-pull-secret\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132134 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-service-ca\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132158 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b6f934-07bb-42c1-8e3d-5bc2369b1716-tmp\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132181 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-oauth-serving-cert\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132220 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132238 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-node-log\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132259 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-metrics-client-ca\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132276 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132306 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-socket-dir-parent\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132338 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-kubelet\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132364 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-agent-certs\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132414 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-tls\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132448 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132485 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnqm\" (UniqueName: \"kubernetes.io/projected/63b6f934-07bb-42c1-8e3d-5bc2369b1716-kube-api-access-8cnqm\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132505 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-kubelet-config\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132522 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132532 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-agent-certs\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:06:05.132629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132543 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkz9n\" (UniqueName: \"kubernetes.io/projected/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-kube-api-access-gkz9n\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.133430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132670 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-tls\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.133430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132683 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-host\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.133430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132711 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-multus\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.133430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132772 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-sys\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.133430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.132876 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63b6f934-07bb-42c1-8e3d-5bc2369b1716-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.148096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148060 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 22:06:05.148096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148056 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.148300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148056 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148145 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148176 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:06:05.148300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148222 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.148300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148231 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:06:05.148300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148255 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148294 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148305 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148314 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148349 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148370 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148417 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148462 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.148546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148493 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.148845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148603 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148626 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:06:05.148845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148647 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:06:05.148845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148655 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148688 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:06:05.148845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148715 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:06:05.148845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148811 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:06:05.149089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148833 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:06:05.149089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.148982 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:06:05.150058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.150034 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.150775 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28191529-9de8-4283-bd79-6b4de730919e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.150893 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a-konnectivity-ca\") pod \"konnectivity-agent-zzcqw\" (UID: \"b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a\") " pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.151155 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.151348 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.151717 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.151929 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-iptables-alerter-script\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.152132 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cni-binary-copy\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.152315 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.152475 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed940156-2f94-4372-a949-f07e265f0c8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.152596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.152518 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 22:06:05.153083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.152720 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-metrics-client-ca\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.158255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.158230 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-serviceca\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:06:05.159646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.159624 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b6f934-07bb-42c1-8e3d-5bc2369b1716-metrics-client-ca\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.159690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.159660 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54880730-d3aa-4d76-8234-0dc766363113-metrics-client-ca\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.163021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.162994 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.184879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.184844 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrhp\" (UniqueName: \"kubernetes.io/projected/6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8-kube-api-access-wsrhp\") pod \"network-metrics-daemon-h7lsc\" (UID: \"6636a0f1-0b17-4bcd-9a9b-6d73e5c558d8\") " pod="openshift-multus/network-metrics-daemon-h7lsc" Nov 02 22:06:05.185034 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.185015 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvddc\" (UniqueName: \"kubernetes.io/projected/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kube-api-access-vvddc\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.185090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.185015 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk29s\" (UniqueName: \"kubernetes.io/projected/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-kube-api-access-dk29s\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:06:05.185090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.185051 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzt5g\" (UniqueName: \"kubernetes.io/projected/306b1470-ddb8-4e0c-b450-3c2ddb541faf-kube-api-access-rzt5g\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.185151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.185093 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ljx\" (UniqueName: \"kubernetes.io/projected/76c3a586-fa6f-42bd-a0d8-42211a9ae53a-kube-api-access-c2ljx\") pod \"dns-default-vxpxh\" (UID: \"76c3a586-fa6f-42bd-a0d8-42211a9ae53a\") " pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.185194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.185177 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwd52\" (UniqueName: \"kubernetes.io/projected/e453e98d-c621-4498-ad81-8cbc7eee10fe-kube-api-access-kwd52\") pod \"network-check-target-pb5sz\" (UID: \"e453e98d-c621-4498-ad81-8cbc7eee10fe\") " pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:06:05.186579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.186551 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6gk\" (UniqueName: \"kubernetes.io/projected/efae3ac8-4b9e-4598-99b0-33eb10cd1935-kube-api-access-sk6gk\") pod \"monitoring-plugin-77f4f76594-fzzfw\" (UID: \"efae3ac8-4b9e-4598-99b0-33eb10cd1935\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:05.187799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.187781 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvfm\" (UniqueName: \"kubernetes.io/projected/28191529-9de8-4283-bd79-6b4de730919e-kube-api-access-5gvfm\") pod \"prometheus-operator-9cd89b74c-rqwlc\" (UID: \"28191529-9de8-4283-bd79-6b4de730919e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rqwlc" Nov 02 22:06:05.188679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.188662 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfv6\" (UniqueName: \"kubernetes.io/projected/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-kube-api-access-hcfv6\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:06:05.188821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.188805 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktm5\" (UniqueName: \"kubernetes.io/projected/d6ed1d10-fcee-4a00-a544-efb9346dac66-kube-api-access-8ktm5\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.189470 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.189450 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jph7w\" (UniqueName: \"kubernetes.io/projected/ed940156-2f94-4372-a949-f07e265f0c8b-kube-api-access-jph7w\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.189674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.189659 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwps\" (UniqueName: \"kubernetes.io/projected/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-kube-api-access-hlwps\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.189777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.189751 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgtj\" (UniqueName: \"kubernetes.io/projected/b1cc496e-e59e-4f60-87bd-eeb2b18a165c-kube-api-access-vlgtj\") pod \"telemeter-client-674c5d5fcb-845r7\" (UID: \"b1cc496e-e59e-4f60-87bd-eeb2b18a165c\") " pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.189991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.189976 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzjj\" (UniqueName: \"kubernetes.io/projected/7ebd138c-3a98-4268-b843-7b18f6ab68f8-kube-api-access-9jzjj\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.190052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.190036 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln994\" (UniqueName: \"kubernetes.io/projected/b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9-kube-api-access-ln994\") pod \"ingress-canary-8zm4n\" (UID: \"b494cbbd-3f65-4fb9-8454-c3d3fd12a5d9\") " pod="openshift-ingress-canary/ingress-canary-8zm4n" Nov 02 22:06:05.191058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.191042 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnqm\" (UniqueName: \"kubernetes.io/projected/63b6f934-07bb-42c1-8e3d-5bc2369b1716-kube-api-access-8cnqm\") pod \"thanos-querier-5bf65b7c55-xx8wq\" (UID: \"63b6f934-07bb-42c1-8e3d-5bc2369b1716\") " pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:05.191671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.191655 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7x7\" (UniqueName: \"kubernetes.io/projected/54880730-d3aa-4d76-8234-0dc766363113-kube-api-access-pc7x7\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.192928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.192912 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgw2f\" (UniqueName: \"kubernetes.io/projected/22f84c42-8ee4-4133-a636-f02598d2b61b-kube-api-access-vgw2f\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:06:05.234020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.233988 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-socket-dir-parent\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234019 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-kubelet\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234036 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-kubelet-config\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.234207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234052 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkz9n\" (UniqueName: \"kubernetes.io/projected/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-kube-api-access-gkz9n\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.234207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234068 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-host\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234111 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-host\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234118 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-kubelet\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234172 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-socket-dir-parent\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234181 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-kubelet-config\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234265 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-multus\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234289 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-sys\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234303 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-host\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234318 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234348 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3ac303-6d4c-4f0a-9279-2b3c428adccd-host\") pod \"node-ca-lsbsf\" (UID: \"cc3ac303-6d4c-4f0a-9279-2b3c428adccd\") " pod="openshift-image-registry/node-ca-lsbsf" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234359 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-sys\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234364 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-multus\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234379 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234386 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-wtmp\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.234419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234415 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-lib-modules\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234436 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-oauth-config\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234456 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-k8s-cni-cncf-io\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234487 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-run\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234509 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-var-lib-kubelet\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234529 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-lib-modules\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234528 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234560 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cnibin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234597 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234601 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-serving-cert\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234606 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-k8s-cni-cncf-io\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234619 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-run\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234626 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234642 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-cnibin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234662 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234618 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-var-lib-kubelet\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234673 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-slash\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234669 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-node-exporter-wtmp\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234705 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-ovn\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234741 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-ovn\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234738 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-trusted-ca-bundle\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234755 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-slash\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234786 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f84c42-8ee4-4133-a636-f02598d2b61b-hosts-file\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234815 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-tmp\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234840 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-systemd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234852 5109 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:06:05.234849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234867 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234876 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22f84c42-8ee4-4133-a636-f02598d2b61b-hosts-file\") pod \"node-resolver-6lwq6\" (UID: \"22f84c42-8ee4-4133-a636-f02598d2b61b\") " pod="openshift-dns/node-resolver-6lwq6" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234890 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-conf-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234917 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234924 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234934 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-systemd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234944 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-device-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234951 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-multus-conf-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234979 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234981 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-modprobe-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.234996 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-device-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235012 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-dbus\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235038 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-config\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235063 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-netns\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235085 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235093 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-modprobe-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235113 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235147 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-bin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235152 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74da9552-2470-4a9f-8160-d3a29aa5d9f5-dbus\") pod \"global-pull-secret-syncer-knp2v\" (UID: \"74da9552-2470-4a9f-8160-d3a29aa5d9f5\") " pod="kube-system/global-pull-secret-syncer-knp2v" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235186 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-hostroot\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235200 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-var-lib-cni-bin\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235198 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-tmp\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235216 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-kubernetes\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235239 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-run-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235251 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-kubelet\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235259 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-kubernetes\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235276 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-systemd-units\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235280 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-run-netns\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235302 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-log-socket\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235325 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-kubelet\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235331 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-cnibin\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235297 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-hostroot\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235335 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-systemd-units\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235372 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-log-socket\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235383 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-cnibin\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235401 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-netd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235424 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysconfig\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235452 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ebd138c-3a98-4268-b843-7b18f6ab68f8-crio-socket\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235470 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-netd\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235480 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235506 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-bin\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235517 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235525 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysconfig\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235534 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-netns\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235586 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-netns\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235587 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-root\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235619 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54880730-d3aa-4d76-8234-0dc766363113-root\") pod \"node-exporter-7sslx\" (UID: \"54880730-d3aa-4d76-8234-0dc766363113\") " pod="openshift-monitoring/node-exporter-7sslx" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235634 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-cni-bin\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235651 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235675 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ebd138c-3a98-4268-b843-7b18f6ab68f8-crio-socket\") pod \"insights-runtime-extractor-p27gn\" (UID: \"7ebd138c-3a98-4268-b843-7b18f6ab68f8\") " pod="openshift-insights/insights-runtime-extractor-p27gn" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235686 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-host-slash\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235713 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed-host-slash\") pod \"iptables-alerter-47zpv\" (UID: \"51d87cc6-0ee0-44bc-9a3e-43ea8fefbfed\") " pod="openshift-network-operator/iptables-alerter-47zpv" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235729 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-systemd\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235753 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-sys\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235766 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-config\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235780 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-conf\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235804 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-etc-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235830 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-systemd\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235833 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-sys\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235829 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-system-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235867 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-etc-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235875 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-system-cni-dir\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235900 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-var-lib-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235925 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-system-cni-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235940 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-trusted-ca-bundle\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235924 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-conf\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235963 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235962 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-var-lib-openvswitch\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235980 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed940156-2f94-4372-a949-f07e265f0c8b-system-cni-dir\") pod \"multus-additional-cni-plugins-55xk7\" (UID: \"ed940156-2f94-4372-a949-f07e265f0c8b\") " pod="openshift-multus/multus-additional-cni-plugins-55xk7" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.235990 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.235993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236013 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-multus-certs\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236071 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-etc-kubernetes\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236074 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236090 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-service-ca\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236092 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/306b1470-ddb8-4e0c-b450-3c2ddb541faf-etc-sysctl-d\") pod \"tuned-n9jzr\" (UID: \"306b1470-ddb8-4e0c-b450-3c2ddb541faf\") " pod="openshift-cluster-node-tuning-operator/tuned-n9jzr" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236115 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-oauth-serving-cert\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236120 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-host-run-multus-certs\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236140 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236140 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b314a49f-8d9b-41f7-81f6-67f6c486f9b4-etc-kubernetes\") pod \"multus-xf2bk\" (UID: \"b314a49f-8d9b-41f7-81f6-67f6c486f9b4\") " pod="openshift-multus/multus-xf2bk" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236180 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-node-log\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236213 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6ed1d10-fcee-4a00-a544-efb9346dac66-node-log\") pod \"ovnkube-node-5xqr8\" (UID: \"d6ed1d10-fcee-4a00-a544-efb9346dac66\") " pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.236215 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7648ae20-ac24-49f7-a8a7-11b204b9a2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-4fkxp\" (UID: \"7648ae20-ac24-49f7-a8a7-11b204b9a2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fkxp" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.237288 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-service-ca\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.237388 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-oauth-serving-cert\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.237535 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-oauth-config\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.238251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.237551 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-serving-cert\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.276965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.276936 5109 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:06:05.277103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.277068 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" Nov 02 22:06:05.292714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.292685 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkz9n\" (UniqueName: \"kubernetes.io/projected/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-kube-api-access-gkz9n\") pod \"console-584dc6f89b-wtdxr\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.590166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.590127 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:05.739494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.739462 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-674c5d5fcb-845r7"] Nov 02 22:06:05.742917 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:05.742890 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cc496e_e59e_4f60_87bd_eeb2b18a165c.slice/crio-cf91b1f276ecb63009e3a06ad3e591a815d17e92770b500e1b5f4b1055cae347 WatchSource:0}: Error finding container cf91b1f276ecb63009e3a06ad3e591a815d17e92770b500e1b5f4b1055cae347: Status 404 returned error can't find the container with id cf91b1f276ecb63009e3a06ad3e591a815d17e92770b500e1b5f4b1055cae347 Nov 02 22:06:05.896431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.896354 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vxpxh" Nov 02 22:06:05.941164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.941122 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" event={"ID":"b1cc496e-e59e-4f60-87bd-eeb2b18a165c","Type":"ContainerStarted","Data":"cf91b1f276ecb63009e3a06ad3e591a815d17e92770b500e1b5f4b1055cae347"} Nov 02 22:06:05.956858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:05.956822 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584dc6f89b-wtdxr"] Nov 02 22:06:05.960293 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:05.960265 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cb5257_ef6c_4ed4_8ef6_3ecfaef5669e.slice/crio-8a02879ef992001b2a904e6bfb9ac42ac41a8f5eb26eb566d11defc058d11573 WatchSource:0}: Error finding container 8a02879ef992001b2a904e6bfb9ac42ac41a8f5eb26eb566d11defc058d11573: Status 404 returned error can't find the container with id 8a02879ef992001b2a904e6bfb9ac42ac41a8f5eb26eb566d11defc058d11573 Nov 02 22:06:06.947805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:06.947762 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" event={"ID":"63b6f934-07bb-42c1-8e3d-5bc2369b1716","Type":"ContainerStarted","Data":"d0bdaa128058df586709c5c3b1a8cd3eb280b3b45f4cd3686022325247ae97a5"} Nov 02 22:06:06.948265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:06.947893 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" event={"ID":"63b6f934-07bb-42c1-8e3d-5bc2369b1716","Type":"ContainerStarted","Data":"dddaf5c41969c37aa8fe6b92b828ebf515a0a1e07abd185886feffa75680d4d0"} Nov 02 22:06:06.948265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:06.947938 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" event={"ID":"63b6f934-07bb-42c1-8e3d-5bc2369b1716","Type":"ContainerStarted","Data":"4839b30425715ea7c790582ed21e9ed7dd9c07f1e6961277ceae0d1d13f1bf3c"} Nov 02 22:06:06.949592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:06.949531 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584dc6f89b-wtdxr" event={"ID":"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e","Type":"ContainerStarted","Data":"8a02879ef992001b2a904e6bfb9ac42ac41a8f5eb26eb566d11defc058d11573"} Nov 02 22:06:06.956826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:06.956790 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:06:07.959743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:07.959606 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" event={"ID":"efae3ac8-4b9e-4598-99b0-33eb10cd1935","Type":"ContainerStarted","Data":"f5c4c6e597ab51b1ede89d9c2217cd7d7fa1d5d3a43b39d076f9e80fe257ccfc"} Nov 02 22:06:08.703517 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:08.703478 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:08.968126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:08.968037 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" event={"ID":"b1cc496e-e59e-4f60-87bd-eeb2b18a165c","Type":"ContainerStarted","Data":"d348c651975129fa67d011ae2ea36a892f45838a5b9c09e6a85828b1a9210942"} Nov 02 22:06:08.971921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:08.971882 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" event={"ID":"63b6f934-07bb-42c1-8e3d-5bc2369b1716","Type":"ContainerStarted","Data":"30b06f9c6ad1741f2e78962a0d00f0f3edc134caf8a3cde365a42bdb61e4634b"} Nov 02 22:06:08.972089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:08.971961 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" event={"ID":"63b6f934-07bb-42c1-8e3d-5bc2369b1716","Type":"ContainerStarted","Data":"7cdf05769af7abdbec24079d1d7f0eaec6fc9ac1bb9e9443e153af12aa9ccf77"} Nov 02 22:06:08.972089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:08.971998 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" event={"ID":"63b6f934-07bb-42c1-8e3d-5bc2369b1716","Type":"ContainerStarted","Data":"160c65366e171d3034152957fa6e59d1291f5261b543365675aa4f192401f1ab"} Nov 02 22:06:08.972219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:08.972092 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:09.977467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:09.977343 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584dc6f89b-wtdxr" event={"ID":"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e","Type":"ContainerStarted","Data":"e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e"} Nov 02 22:06:10.091097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:10.091034 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-584dc6f89b-wtdxr" podStartSLOduration=2.3666261 podStartE2EDuration="6.091019419s" totalImagesPullingTime="3.724393319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:10.090915759 +0000 UTC m=+6.355908979" watchObservedRunningTime="2026-11-02 22:06:10.091019419 +0000 UTC m=+6.356012615" Nov 02 22:06:10.982896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:10.982783 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" event={"ID":"b1cc496e-e59e-4f60-87bd-eeb2b18a165c","Type":"ContainerStarted","Data":"f6ccfb19f04e91b58a12454b1b30df87887244b56caac5d47c285ad96ff52886"} Nov 02 22:06:10.982896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:10.982864 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-674c5d5fcb-845r7" event={"ID":"b1cc496e-e59e-4f60-87bd-eeb2b18a165c","Type":"ContainerStarted","Data":"d0014a01179e9d481a1bf923c8fe75a95692d42f93c6b59458a2ea665e9acfe1"} Nov 02 22:06:11.649340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:11.649256 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:11.649494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:11.649433 5109 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:06:11.654217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:11.654191 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fzzfw" Nov 02 22:06:13.076501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:13.076461 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:13.860945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:13.860906 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584dc6f89b-wtdxr"] Nov 02 22:06:14.509277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:14.509239 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zzcqw" Nov 02 22:06:14.984798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:14.984719 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bf65b7c55-xx8wq" Nov 02 22:06:15.023373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:15.023344 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:15.023619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:15.023606 5109 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:06:26.490155 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:26.490114 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:06:31.625643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:31.625603 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5xqr8" Nov 02 22:06:36.959881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:36.959845 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pb5sz" Nov 02 22:06:39.019127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.019087 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-584dc6f89b-wtdxr" podUID="26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" containerName="console" containerID="cri-o://e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e" gracePeriod=15 Nov 02 22:06:39.269836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.269774 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:39.318001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.317961 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-service-ca\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318010 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-tmp\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318071 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-kube-api-access-gkz9n\" (UniqueName: \"kubernetes.io/projected/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-kube-api-access-gkz9n\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318112 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-config\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-config\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318151 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-trusted-ca-bundle\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318201 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-serving-cert\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318236 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-oauth-config\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318266 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-oauth-serving-cert\") pod \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\" (UID: \"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e\") " Nov 02 22:06:39.318694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318434 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-tmp" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:06:39.318694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.318497 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-service-ca" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:39.319198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.319168 5109 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.319303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.319203 5109 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-tmp\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.319534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.319470 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-oauth-serving-cert" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:39.319732 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.319704 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-config" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:39.319809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.319721 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-trusted-ca-bundle" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:39.320959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.320932 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-oauth-config" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.321059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.321035 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-kube-api-access-gkz9n" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "kube-api-access-gkz9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:39.326036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.326008 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-serving-cert" pod "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" (UID: "26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.420312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.420258 5109 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-config\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.420312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.420309 5109 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-trusted-ca-bundle\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.420312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.420321 5109 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-serving-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.420312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.420330 5109 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-console-oauth-config\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.420614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.420338 5109 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-oauth-serving-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.420614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.420347 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkz9n\" (UniqueName: \"kubernetes.io/projected/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e-kube-api-access-gkz9n\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.669245 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.669161 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:39.669453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.669440 5109 cpu_manager.go:414] "containerMap: removing container" podUID="26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" containerName="console" Nov 02 22:06:39.669495 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.669455 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" containerName="console" Nov 02 22:06:39.669528 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.669513 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" containerName="console" Nov 02 22:06:39.672311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.672294 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722053 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce420e46-db02-4284-8adf-7db89e3fe8f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722089 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce420e46-db02-4284-8adf-7db89e3fe8f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722105 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce420e46-db02-4284-8adf-7db89e3fe8f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722177 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce420e46-db02-4284-8adf-7db89e3fe8f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722222 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce420e46-db02-4284-8adf-7db89e3fe8f0-config-out\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722281 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-web-config\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722306 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722363 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722410 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722426 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722447 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pq8\" (UniqueName: \"kubernetes.io/projected/ce420e46-db02-4284-8adf-7db89e3fe8f0-kube-api-access-h6pq8\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722473 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.722617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.722557 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.768685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.768648 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 22:06:39.768685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.768648 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 22:06:39.768685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.768662 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 22:06:39.769022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.768648 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 22:06:39.769022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.768731 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 22:06:39.769022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.768861 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 22:06:39.769185 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.769044 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 22:06:39.769185 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.769111 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6jjrn\"" Nov 02 22:06:39.815162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.815124 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 22:06:39.815718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.815703 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 22:06:39.816509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.816489 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:39.823269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823242 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce420e46-db02-4284-8adf-7db89e3fe8f0-config-out\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823277 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-web-config\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823299 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823318 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823348 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823370 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823394 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pq8\" (UniqueName: \"kubernetes.io/projected/ce420e46-db02-4284-8adf-7db89e3fe8f0-kube-api-access-h6pq8\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823431 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823465 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823500 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce420e46-db02-4284-8adf-7db89e3fe8f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823531 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce420e46-db02-4284-8adf-7db89e3fe8f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823555 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce420e46-db02-4284-8adf-7db89e3fe8f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.823860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823605 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce420e46-db02-4284-8adf-7db89e3fe8f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.824011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.823986 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce420e46-db02-4284-8adf-7db89e3fe8f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.824910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.824885 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce420e46-db02-4284-8adf-7db89e3fe8f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.825413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.825385 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce420e46-db02-4284-8adf-7db89e3fe8f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.826290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.826263 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce420e46-db02-4284-8adf-7db89e3fe8f0-config-out\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.826545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.826527 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.826964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.826947 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.826964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.826957 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce420e46-db02-4284-8adf-7db89e3fe8f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.827085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.827050 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.827157 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.827134 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-web-config\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.827948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.827917 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.828041 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.827993 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.828303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.828285 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce420e46-db02-4284-8adf-7db89e3fe8f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.881828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.881794 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pq8\" (UniqueName: \"kubernetes.io/projected/ce420e46-db02-4284-8adf-7db89e3fe8f0-kube-api-access-h6pq8\") pod \"alertmanager-main-0\" (UID: \"ce420e46-db02-4284-8adf-7db89e3fe8f0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.990919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:39.990831 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:40.083123 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.083087 5109 generic.go:356] "Generic (PLEG): container finished" podID="26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" containerID="e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e" exitCode=0 Nov 02 22:06:40.083547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.083147 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584dc6f89b-wtdxr" Nov 02 22:06:40.083547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.083176 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584dc6f89b-wtdxr" event={"ID":"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e","Type":"ContainerDied","Data":"e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e"} Nov 02 22:06:40.083547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.083232 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584dc6f89b-wtdxr" event={"ID":"26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e","Type":"ContainerDied","Data":"8a02879ef992001b2a904e6bfb9ac42ac41a8f5eb26eb566d11defc058d11573"} Nov 02 22:06:40.083547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.083268 5109 scope.go:122] "RemoveContainer" containerID="e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e" Nov 02 22:06:40.091268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.091250 5109 scope.go:122] "RemoveContainer" containerID="e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e" Nov 02 22:06:40.091537 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:40.091518 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e\": container with ID starting with e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e not found: ID does not exist" containerID="e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e" Nov 02 22:06:40.091621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.091545 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e"} err="failed to get container status \"e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e\": rpc error: code = NotFound desc = could not find container \"e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e\": container with ID starting with e754c1e798e40548611fd1cda216d6d941c718024da67c0daa4e7f7d351f0b9e not found: ID does not exist" Nov 02 22:06:40.236810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.236776 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584dc6f89b-wtdxr"] Nov 02 22:06:40.240765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.240737 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-584dc6f89b-wtdxr"] Nov 02 22:06:40.408478 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:40.408445 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:40.411474 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:06:40.411443 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce420e46_db02_4284_8adf_7db89e3fe8f0.slice/crio-6468f9a98b42a8215f3afac6781cdd8ce752fb3eb975a20d2d3259b5709dfbf5 WatchSource:0}: Error finding container 6468f9a98b42a8215f3afac6781cdd8ce752fb3eb975a20d2d3259b5709dfbf5: Status 404 returned error can't find the container with id 6468f9a98b42a8215f3afac6781cdd8ce752fb3eb975a20d2d3259b5709dfbf5 Nov 02 22:06:41.087833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:41.087798 5109 generic.go:356] "Generic (PLEG): container finished" podID="ce420e46-db02-4284-8adf-7db89e3fe8f0" containerID="3ffbf447827b81f2100a2b2085bd94f9c019eb31ec993b02b22bab195a47bdf2" exitCode=0 Nov 02 22:06:41.088279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:41.087884 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerDied","Data":"3ffbf447827b81f2100a2b2085bd94f9c019eb31ec993b02b22bab195a47bdf2"} Nov 02 22:06:41.088279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:41.087949 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerStarted","Data":"6468f9a98b42a8215f3afac6781cdd8ce752fb3eb975a20d2d3259b5709dfbf5"} Nov 02 22:06:41.889743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:41.889705 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e" path="/var/lib/kubelet/pods/26cb5257-ef6c-4ed4-8ef6-3ecfaef5669e/volumes" Nov 02 22:06:44.103562 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:44.103528 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerStarted","Data":"0aad8df486cfb2383afaa39bfebafeadc7ab7b047a3b2fa2bffba784a1e89d81"} Nov 02 22:06:44.103986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:44.103657 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerStarted","Data":"eeebd0be65aaadef86e3e962b02147ac93d04873b40a4de4f2ed1a53c72f5fa3"} Nov 02 22:06:44.103986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:44.103692 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerStarted","Data":"d506dc5832a19baa008a302bc2d23777a38967cdad09f4cfa74f54d0312f3e1c"} Nov 02 22:06:44.103986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:44.103727 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerStarted","Data":"6037688672bcbd58b9c3d188d69d6fa9a9066869b6f43d47ffffc88e0ad5b088"} Nov 02 22:06:44.103986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:44.103771 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerStarted","Data":"24a2f25849b18e0b57592d30087d838e9ed616b15aa1c5ec13864821095f03d0"} Nov 02 22:06:44.103986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:44.103803 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce420e46-db02-4284-8adf-7db89e3fe8f0","Type":"ContainerStarted","Data":"1c32845592ad22ed94b821a782945fbab69cef628652af81c57037b383dd5395"} Nov 02 22:06:44.366821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:06:44.366714 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.397122327 podStartE2EDuration="5.366700818s" totalImagesPullingTime="1.956701642s" totalInitContainerRuntime="12.876849ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:44.341390521 +0000 UTC m=+40.606383741" watchObservedRunningTime="2026-11-02 22:06:44.366700818 +0000 UTC m=+40.631694033" Nov 02 22:06:56.489087 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:06:56.489037 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:07:26.490303 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:07:26.490262 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:07:56.488745 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:07:56.488706 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:08:26.489180 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:08:26.489140 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:08:56.490110 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:08:56.490071 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:08:59.136469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.136435 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c"] Nov 02 22:08:59.139020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.139003 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.189111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.189070 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 22:08:59.189295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.189132 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 22:08:59.191305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.191284 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.191356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.191333 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.191440 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.191421 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vghq\" (UniqueName: \"kubernetes.io/projected/7c88fd00-63fa-4815-87b7-345466ec99ec-kube-api-access-7vghq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.235978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.235933 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-x8rmm\"" Nov 02 22:08:59.236655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.236630 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c"] Nov 02 22:08:59.273638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.273588 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt"] Nov 02 22:08:59.276141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.276114 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.292737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.292687 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.292737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.292744 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlrs\" (UniqueName: \"kubernetes.io/projected/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-kube-api-access-shlrs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.292944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.292767 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.292944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.292804 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.292944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.292822 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vghq\" (UniqueName: \"kubernetes.io/projected/7c88fd00-63fa-4815-87b7-345466ec99ec-kube-api-access-7vghq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.292944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.292840 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.293128 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.293066 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.293128 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.293101 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.333414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.333374 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt"] Nov 02 22:08:59.351325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.351287 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vghq\" (UniqueName: \"kubernetes.io/projected/7c88fd00-63fa-4815-87b7-345466ec99ec-kube-api-access-7vghq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.394205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.394123 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.394205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.394192 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlrs\" (UniqueName: \"kubernetes.io/projected/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-kube-api-access-shlrs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.394429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.394221 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.394619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.394599 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.394667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.394613 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.397339 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.397323 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-lhv59\"" Nov 02 22:08:59.451227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.451194 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:08:59.453208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.453165 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlrs\" (UniqueName: \"kubernetes.io/projected/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-kube-api-access-shlrs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.589182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.589147 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:08:59.655600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.655155 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz"] Nov 02 22:08:59.657796 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.657773 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.696168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.696130 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68ca43d-8fcb-4815-af29-a4ed75f36c75-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.696168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.696174 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.696380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.696203 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.696380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.696307 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.696380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.696339 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grc57\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-kube-api-access-grc57\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.696380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.696364 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.696504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.696442 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.708014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.707987 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 22:08:59.708196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.708176 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:08:59.708272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.708226 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:08:59.708272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.708264 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 22:08:59.708353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.708270 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 22:08:59.708353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.708293 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p7cs8\"" Nov 02 22:08:59.708353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.708308 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:08:59.723544 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.723496 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz"] Nov 02 22:08:59.797859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.797817 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68ca43d-8fcb-4815-af29-a4ed75f36c75-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.797859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.797860 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.798127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.797880 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.798127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.797899 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.798127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.797916 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grc57\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-kube-api-access-grc57\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.798127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.798038 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.798127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.798091 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.798771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.798690 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.800815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.800789 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68ca43d-8fcb-4815-af29-a4ed75f36c75-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.800909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.800894 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.800978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.800963 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.801013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.800963 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.829078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.829049 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c"] Nov 02 22:08:59.832208 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:08:59.832183 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c88fd00_63fa_4815_87b7_345466ec99ec.slice/crio-8ac1618a9b2b0ad9fdf37d0ebc1e4e4174522a9ecfc606fcfe81c37a462e3a0a WatchSource:0}: Error finding container 8ac1618a9b2b0ad9fdf37d0ebc1e4e4174522a9ecfc606fcfe81c37a462e3a0a: Status 404 returned error can't find the container with id 8ac1618a9b2b0ad9fdf37d0ebc1e4e4174522a9ecfc606fcfe81c37a462e3a0a Nov 02 22:08:59.856740 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.856696 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.856905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.856838 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grc57\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-kube-api-access-grc57\") pod \"istiod-openshift-gateway-7876bfcb74-l24hz\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:08:59.957456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.957380 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt"] Nov 02 22:08:59.960343 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:08:59.960316 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c6328d_5fa5_4c30_9e91_2c1a813a5c3b.slice/crio-d2d9c28d8182395ae1ee12500c9b54e5cc49e5582b1ade8e903660e1e5354349 WatchSource:0}: Error finding container d2d9c28d8182395ae1ee12500c9b54e5cc49e5582b1ade8e903660e1e5354349: Status 404 returned error can't find the container with id d2d9c28d8182395ae1ee12500c9b54e5cc49e5582b1ade8e903660e1e5354349 Nov 02 22:08:59.970102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:08:59.970072 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:09:00.338623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:00.338599 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz"] Nov 02 22:09:00.341055 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:09:00.341032 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68ca43d_8fcb_4815_af29_a4ed75f36c75.slice/crio-978c66146d53b31ad494991d3f60949038a3b515790c2bcbb5bd7e0a459afb03 WatchSource:0}: Error finding container 978c66146d53b31ad494991d3f60949038a3b515790c2bcbb5bd7e0a459afb03: Status 404 returned error can't find the container with id 978c66146d53b31ad494991d3f60949038a3b515790c2bcbb5bd7e0a459afb03 Nov 02 22:09:00.562777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:00.562744 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" event={"ID":"f68ca43d-8fcb-4815-af29-a4ed75f36c75","Type":"ContainerStarted","Data":"978c66146d53b31ad494991d3f60949038a3b515790c2bcbb5bd7e0a459afb03"} Nov 02 22:09:00.564028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:00.563996 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" event={"ID":"7c88fd00-63fa-4815-87b7-345466ec99ec","Type":"ContainerStarted","Data":"8ac1618a9b2b0ad9fdf37d0ebc1e4e4174522a9ecfc606fcfe81c37a462e3a0a"} Nov 02 22:09:00.565108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:00.565086 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" event={"ID":"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b","Type":"ContainerStarted","Data":"d2d9c28d8182395ae1ee12500c9b54e5cc49e5582b1ade8e903660e1e5354349"} Nov 02 22:09:05.074642 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:05.074602 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:09:05.075059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:05.074689 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:09:05.590686 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:05.590646 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" event={"ID":"f68ca43d-8fcb-4815-af29-a4ed75f36c75","Type":"ContainerStarted","Data":"b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c"} Nov 02 22:09:05.590915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:05.590773 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:09:05.799381 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:05.799307 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" podStartSLOduration=2.0689662540000002 podStartE2EDuration="6.799292876s" totalImagesPullingTime="4.730326622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:05.752090604 +0000 UTC m=+182.017083825" watchObservedRunningTime="2026-11-02 22:09:05.799292876 +0000 UTC m=+182.064286096" Nov 02 22:09:06.596859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:06.596795 5109 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-l24hz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:09:06.597364 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:06.596890 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" podUID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:09:07.601386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:07.601351 5109 generic.go:356] "Generic (PLEG): container finished" podID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerID="63c20dbeaa4b5afdd9a2fc0942bdddab649d42b57c19945bf8496b606437fb66" exitCode=0 Nov 02 22:09:07.601816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:07.601439 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" event={"ID":"7c88fd00-63fa-4815-87b7-345466ec99ec","Type":"ContainerDied","Data":"63c20dbeaa4b5afdd9a2fc0942bdddab649d42b57c19945bf8496b606437fb66"} Nov 02 22:09:08.606917 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:08.606884 5109 generic.go:356] "Generic (PLEG): container finished" podID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerID="3847bda489e4c570a2bb67b3f82a553fc9a803b913a7d30aee344cb60a1c98a4" exitCode=0 Nov 02 22:09:08.607400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:08.606945 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" event={"ID":"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b","Type":"ContainerDied","Data":"3847bda489e4c570a2bb67b3f82a553fc9a803b913a7d30aee344cb60a1c98a4"} Nov 02 22:09:09.596513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:09.596483 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:09:10.617949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:10.617914 5109 generic.go:356] "Generic (PLEG): container finished" podID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerID="18f9a63e26e5842bbf7436bec79682d8384930e8194c5771f829fcc5999436ef" exitCode=0 Nov 02 22:09:10.618403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:10.618019 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" event={"ID":"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b","Type":"ContainerDied","Data":"18f9a63e26e5842bbf7436bec79682d8384930e8194c5771f829fcc5999436ef"} Nov 02 22:09:11.623152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:11.623116 5109 generic.go:356] "Generic (PLEG): container finished" podID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerID="0fe88187a2a289760fb91102fe5751e3e3c26efb5ed4a90a7e1a4fddca884c5e" exitCode=0 Nov 02 22:09:11.623152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:11.623153 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" event={"ID":"7c88fd00-63fa-4815-87b7-345466ec99ec","Type":"ContainerDied","Data":"0fe88187a2a289760fb91102fe5751e3e3c26efb5ed4a90a7e1a4fddca884c5e"} Nov 02 22:09:18.662635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:18.662592 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" event={"ID":"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b","Type":"ContainerStarted","Data":"f06df4adcc288fb541e0e93fbaff2cc5f0a8fc8f0533553c357c7e8ca24b442e"} Nov 02 22:09:18.774547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:18.774465 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" podStartSLOduration=2.279874934 podStartE2EDuration="19.774444667s" totalImagesPullingTime="17.494569733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:18.773598239 +0000 UTC m=+195.038591460" watchObservedRunningTime="2026-11-02 22:09:18.774444667 +0000 UTC m=+195.039437881" Nov 02 22:09:19.668098 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:19.668011 5109 generic.go:356] "Generic (PLEG): container finished" podID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerID="605ad65856fb57dc82f7e6cec61a0a74169e2f71622a75175a104a9e9ec9637b" exitCode=0 Nov 02 22:09:19.668514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:19.668093 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" event={"ID":"7c88fd00-63fa-4815-87b7-345466ec99ec","Type":"ContainerDied","Data":"605ad65856fb57dc82f7e6cec61a0a74169e2f71622a75175a104a9e9ec9637b"} Nov 02 22:09:19.670158 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:19.670130 5109 generic.go:356] "Generic (PLEG): container finished" podID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerID="f06df4adcc288fb541e0e93fbaff2cc5f0a8fc8f0533553c357c7e8ca24b442e" exitCode=0 Nov 02 22:09:19.670277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:19.670171 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" event={"ID":"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b","Type":"ContainerDied","Data":"f06df4adcc288fb541e0e93fbaff2cc5f0a8fc8f0533553c357c7e8ca24b442e"} Nov 02 22:09:20.824846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:20.824814 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:09:20.828264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:20.828235 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:09:21.014094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.014066 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-util\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-util\") pod \"7c88fd00-63fa-4815-87b7-345466ec99ec\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " Nov 02 22:09:21.014255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.014123 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-bundle\") pod \"7c88fd00-63fa-4815-87b7-345466ec99ec\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " Nov 02 22:09:21.014255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.014169 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-kube-api-access-shlrs\" (UniqueName: \"kubernetes.io/projected/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-kube-api-access-shlrs\") pod \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " Nov 02 22:09:21.014255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.014204 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c88fd00-63fa-4815-87b7-345466ec99ec-kube-api-access-7vghq\" (UniqueName: \"kubernetes.io/projected/7c88fd00-63fa-4815-87b7-345466ec99ec-kube-api-access-7vghq\") pod \"7c88fd00-63fa-4815-87b7-345466ec99ec\" (UID: \"7c88fd00-63fa-4815-87b7-345466ec99ec\") " Nov 02 22:09:21.014415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.014259 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-util\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-util\") pod \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " Nov 02 22:09:21.014415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.014305 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-bundle\") pod \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\" (UID: \"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b\") " Nov 02 22:09:21.014724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.014624 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-bundle" pod "7c88fd00-63fa-4815-87b7-345466ec99ec" (UID: "7c88fd00-63fa-4815-87b7-345466ec99ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:21.015092 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.015069 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-bundle" pod "68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" (UID: "68c6328d-5fa5-4c30-9e91-2c1a813a5c3b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:21.016756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.016705 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c88fd00-63fa-4815-87b7-345466ec99ec-kube-api-access-7vghq" pod "7c88fd00-63fa-4815-87b7-345466ec99ec" (UID: "7c88fd00-63fa-4815-87b7-345466ec99ec"). InnerVolumeSpecName "kube-api-access-7vghq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:21.017079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.017061 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-kube-api-access-shlrs" pod "68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" (UID: "68c6328d-5fa5-4c30-9e91-2c1a813a5c3b"). InnerVolumeSpecName "kube-api-access-shlrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:21.019490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.019464 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-util" pod "7c88fd00-63fa-4815-87b7-345466ec99ec" (UID: "7c88fd00-63fa-4815-87b7-345466ec99ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:21.020287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.020263 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-util" pod "68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" (UID: "68c6328d-5fa5-4c30-9e91-2c1a813a5c3b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:21.115773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.115735 5109 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-bundle\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:21.115773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.115769 5109 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-util\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:21.115773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.115778 5109 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c88fd00-63fa-4815-87b7-345466ec99ec-bundle\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:21.116011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.115787 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlrs\" (UniqueName: \"kubernetes.io/projected/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-kube-api-access-shlrs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:21.116011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.115797 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vghq\" (UniqueName: \"kubernetes.io/projected/7c88fd00-63fa-4815-87b7-345466ec99ec-kube-api-access-7vghq\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:21.116011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.115806 5109 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68c6328d-5fa5-4c30-9e91-2c1a813a5c3b-util\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:21.678926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.678898 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" Nov 02 22:09:21.679129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.678897 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448kv2c" event={"ID":"7c88fd00-63fa-4815-87b7-345466ec99ec","Type":"ContainerDied","Data":"8ac1618a9b2b0ad9fdf37d0ebc1e4e4174522a9ecfc606fcfe81c37a462e3a0a"} Nov 02 22:09:21.679129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.679022 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ac1618a9b2b0ad9fdf37d0ebc1e4e4174522a9ecfc606fcfe81c37a462e3a0a" Nov 02 22:09:21.680915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.680888 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" event={"ID":"68c6328d-5fa5-4c30-9e91-2c1a813a5c3b","Type":"ContainerDied","Data":"d2d9c28d8182395ae1ee12500c9b54e5cc49e5582b1ade8e903660e1e5354349"} Nov 02 22:09:21.681030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.680926 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tgrjt" Nov 02 22:09:21.681030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:09:21.680934 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d9c28d8182395ae1ee12500c9b54e5cc49e5582b1ade8e903660e1e5354349" Nov 02 22:09:26.488707 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:09:26.488670 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:09:56.488755 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:09:56.488714 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:10:09.616992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.616957 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f"] Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617248 5109 cpu_manager.go:414] "containerMap: removing container" podUID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerName="pull" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617259 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerName="pull" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617272 5109 cpu_manager.go:414] "containerMap: removing container" podUID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerName="util" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617277 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerName="util" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617289 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerName="util" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617294 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerName="util" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617303 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerName="pull" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617308 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerName="pull" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617315 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerName="extract" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617319 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerName="extract" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617329 5109 cpu_manager.go:414] "containerMap: removing container" podUID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerName="extract" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617333 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerName="extract" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617419 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c88fd00-63fa-4815-87b7-345466ec99ec" containerName="extract" Nov 02 22:10:09.617545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.617428 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c6328d-5fa5-4c30-9e91-2c1a813a5c3b" containerName="extract" Nov 02 22:10:09.619526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.619509 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.687707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.687662 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzzh\" (UniqueName: \"kubernetes.io/projected/cab17d5c-429e-47c2-8272-9d3a7e3dc161-kube-api-access-zqzzh\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.687707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.687704 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-webhook-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.687925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.687820 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cab17d5c-429e-47c2-8272-9d3a7e3dc161-manifests\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.687925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.687855 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-metrics-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.687995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.687959 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.688029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.687995 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cab17d5c-429e-47c2-8272-9d3a7e3dc161-tmp\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.738443 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.738394 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:10:09.738443 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.738445 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:10:09.738737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.738506 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kldsf\"" Nov 02 22:10:09.738737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.738634 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:10:09.739261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.739243 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:10:09.739313 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.739283 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:10:09.788888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.788849 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cab17d5c-429e-47c2-8272-9d3a7e3dc161-tmp\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.788888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.788889 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzzh\" (UniqueName: \"kubernetes.io/projected/cab17d5c-429e-47c2-8272-9d3a7e3dc161-kube-api-access-zqzzh\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.789131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.788910 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-webhook-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.789131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.788956 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cab17d5c-429e-47c2-8272-9d3a7e3dc161-manifests\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.789131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.788976 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-metrics-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.789131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.789015 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.789458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.789436 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cab17d5c-429e-47c2-8272-9d3a7e3dc161-manifests\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.791496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.791473 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cab17d5c-429e-47c2-8272-9d3a7e3dc161-tmp\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.791708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.791684 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.791973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.791954 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-webhook-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.792050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.792032 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cab17d5c-429e-47c2-8272-9d3a7e3dc161-metrics-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.856981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.856951 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzzh\" (UniqueName: \"kubernetes.io/projected/cab17d5c-429e-47c2-8272-9d3a7e3dc161-kube-api-access-zqzzh\") pod \"opendatahub-operator-controller-manager-d99fd484b-npb4f\" (UID: \"cab17d5c-429e-47c2-8272-9d3a7e3dc161\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:09.861095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.861062 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f"] Nov 02 22:10:09.933771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:09.933679 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:10.473795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:10.473743 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f"] Nov 02 22:10:10.476648 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:10:10.476615 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab17d5c_429e_47c2_8272_9d3a7e3dc161.slice/crio-25781abba4602106bf96a9d093208c12d8090e9a7cb4bd31f2400a8e1af4c309 WatchSource:0}: Error finding container 25781abba4602106bf96a9d093208c12d8090e9a7cb4bd31f2400a8e1af4c309: Status 404 returned error can't find the container with id 25781abba4602106bf96a9d093208c12d8090e9a7cb4bd31f2400a8e1af4c309 Nov 02 22:10:10.858907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:10.858868 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" event={"ID":"cab17d5c-429e-47c2-8272-9d3a7e3dc161","Type":"ContainerStarted","Data":"25781abba4602106bf96a9d093208c12d8090e9a7cb4bd31f2400a8e1af4c309"} Nov 02 22:10:14.877369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:14.877291 5109 generic.go:356] "Generic (PLEG): container finished" podID="cab17d5c-429e-47c2-8272-9d3a7e3dc161" containerID="261d0545f591ab8a2652e1f3c383b80c9bdc7e77b90d958e959e26df0a098333" exitCode=0 Nov 02 22:10:14.877768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:14.877382 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" event={"ID":"cab17d5c-429e-47c2-8272-9d3a7e3dc161","Type":"ContainerDied","Data":"261d0545f591ab8a2652e1f3c383b80c9bdc7e77b90d958e959e26df0a098333"} Nov 02 22:10:15.882647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:15.882610 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" event={"ID":"cab17d5c-429e-47c2-8272-9d3a7e3dc161","Type":"ContainerStarted","Data":"69ebd695daf21e04403f036bcab7aee0d13d873d614f755ecd4c3632fa85cbb1"} Nov 02 22:10:15.883057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:15.882720 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:16.067105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:16.067044 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" podStartSLOduration=2.207397043 podStartE2EDuration="7.067028555s" totalImagesPullingTime="4.443041827s" totalInitContainerRuntime="416.589685ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:16.065718413 +0000 UTC m=+252.330711634" watchObservedRunningTime="2026-11-02 22:10:16.067028555 +0000 UTC m=+252.332021773" Nov 02 22:10:18.594244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.594199 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-n9fp6"] Nov 02 22:10:18.596774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.596750 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:18.674401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.674361 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xq9h\" (UniqueName: \"kubernetes.io/projected/adb68bf9-d4b9-4a71-95a3-fd7a917db522-kube-api-access-9xq9h\") pod \"cert-manager-597b95cb84-n9fp6\" (UID: \"adb68bf9-d4b9-4a71-95a3-fd7a917db522\") " pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:18.674597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.674500 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adb68bf9-d4b9-4a71-95a3-fd7a917db522-bound-sa-token\") pod \"cert-manager-597b95cb84-n9fp6\" (UID: \"adb68bf9-d4b9-4a71-95a3-fd7a917db522\") " pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:18.705335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.705290 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 22:10:18.705335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.705304 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 22:10:18.705598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.705440 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wjgv8\"" Nov 02 22:10:18.706403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.706385 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-n9fp6"] Nov 02 22:10:18.775557 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.775518 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xq9h\" (UniqueName: \"kubernetes.io/projected/adb68bf9-d4b9-4a71-95a3-fd7a917db522-kube-api-access-9xq9h\") pod \"cert-manager-597b95cb84-n9fp6\" (UID: \"adb68bf9-d4b9-4a71-95a3-fd7a917db522\") " pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:18.775765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.775625 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adb68bf9-d4b9-4a71-95a3-fd7a917db522-bound-sa-token\") pod \"cert-manager-597b95cb84-n9fp6\" (UID: \"adb68bf9-d4b9-4a71-95a3-fd7a917db522\") " pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:18.834169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.834125 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adb68bf9-d4b9-4a71-95a3-fd7a917db522-bound-sa-token\") pod \"cert-manager-597b95cb84-n9fp6\" (UID: \"adb68bf9-d4b9-4a71-95a3-fd7a917db522\") " pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:18.834335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.834207 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xq9h\" (UniqueName: \"kubernetes.io/projected/adb68bf9-d4b9-4a71-95a3-fd7a917db522-kube-api-access-9xq9h\") pod \"cert-manager-597b95cb84-n9fp6\" (UID: \"adb68bf9-d4b9-4a71-95a3-fd7a917db522\") " pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:18.919707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:18.919613 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-n9fp6" Nov 02 22:10:19.282656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:19.282621 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-n9fp6"] Nov 02 22:10:19.285927 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:10:19.285898 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadb68bf9_d4b9_4a71_95a3_fd7a917db522.slice/crio-31d902def5c9ebf1bb6e4148d932658f6b4d6c23eb077c5cca8c9e3a622e4664 WatchSource:0}: Error finding container 31d902def5c9ebf1bb6e4148d932658f6b4d6c23eb077c5cca8c9e3a622e4664: Status 404 returned error can't find the container with id 31d902def5c9ebf1bb6e4148d932658f6b4d6c23eb077c5cca8c9e3a622e4664 Nov 02 22:10:19.897502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:19.897467 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-n9fp6" event={"ID":"adb68bf9-d4b9-4a71-95a3-fd7a917db522","Type":"ContainerStarted","Data":"31d902def5c9ebf1bb6e4148d932658f6b4d6c23eb077c5cca8c9e3a622e4664"} Nov 02 22:10:24.920021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:24.919981 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-n9fp6" event={"ID":"adb68bf9-d4b9-4a71-95a3-fd7a917db522","Type":"ContainerStarted","Data":"18cfc0dc324c011942f12844cba8a95e70d37579639586fe4ecba2ac1ebe5a53"} Nov 02 22:10:25.031716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.031664 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-n9fp6" podStartSLOduration=2.258580421 podStartE2EDuration="7.031646469s" totalImagesPullingTime="4.773066048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:25.030950987 +0000 UTC m=+261.295944205" watchObservedRunningTime="2026-11-02 22:10:25.031646469 +0000 UTC m=+261.296639689" Nov 02 22:10:25.051542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.051507 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw"] Nov 02 22:10:25.054810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.054787 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.104078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.104037 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qkcwg\"" Nov 02 22:10:25.104290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.104226 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 22:10:25.104290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.104280 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:10:25.104386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.104346 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 22:10:25.104386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.104354 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:10:25.131804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.131769 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/59f64123-31a0-4e5a-bf58-d6232e269329-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.131982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.131889 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59f64123-31a0-4e5a-bf58-d6232e269329-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.131982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.131951 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knn8d\" (UniqueName: \"kubernetes.io/projected/59f64123-31a0-4e5a-bf58-d6232e269329-kube-api-access-knn8d\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.138465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.138432 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw"] Nov 02 22:10:25.232899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.232805 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59f64123-31a0-4e5a-bf58-d6232e269329-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.232899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.232853 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knn8d\" (UniqueName: \"kubernetes.io/projected/59f64123-31a0-4e5a-bf58-d6232e269329-kube-api-access-knn8d\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.232899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.232881 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/59f64123-31a0-4e5a-bf58-d6232e269329-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.233642 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.233616 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59f64123-31a0-4e5a-bf58-d6232e269329-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.235371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.235351 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/59f64123-31a0-4e5a-bf58-d6232e269329-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.291251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.291209 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knn8d\" (UniqueName: \"kubernetes.io/projected/59f64123-31a0-4e5a-bf58-d6232e269329-kube-api-access-knn8d\") pod \"kuadrant-console-plugin-66c4fb575d-djhzw\" (UID: \"59f64123-31a0-4e5a-bf58-d6232e269329\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.367306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.367269 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" Nov 02 22:10:25.730781 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.730757 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw"] Nov 02 22:10:25.733065 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:10:25.733039 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59f64123_31a0_4e5a_bf58_d6232e269329.slice/crio-34d7d3c75f99ce1dcf0c83828208d2ba50bef18d70775a18c32a94c26f95f388 WatchSource:0}: Error finding container 34d7d3c75f99ce1dcf0c83828208d2ba50bef18d70775a18c32a94c26f95f388: Status 404 returned error can't find the container with id 34d7d3c75f99ce1dcf0c83828208d2ba50bef18d70775a18c32a94c26f95f388 Nov 02 22:10:25.925643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:25.925602 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" event={"ID":"59f64123-31a0-4e5a-bf58-d6232e269329","Type":"ContainerStarted","Data":"34d7d3c75f99ce1dcf0c83828208d2ba50bef18d70775a18c32a94c26f95f388"} Nov 02 22:10:26.490079 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:10:26.490041 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:10:26.888636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:26.888605 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-npb4f" Nov 02 22:10:31.956762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:31.956668 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" event={"ID":"59f64123-31a0-4e5a-bf58-d6232e269329","Type":"ContainerStarted","Data":"a040fb9b6a3fbd61fe9063b0b184d8427be3f2a3605c74bc52a5295a146bb654"} Nov 02 22:10:32.066407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:32.066347 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-djhzw" podStartSLOduration=1.125572441 podStartE2EDuration="7.066330996s" totalImagesPullingTime="5.940758555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:32.065303218 +0000 UTC m=+268.330296439" watchObservedRunningTime="2026-11-02 22:10:32.066330996 +0000 UTC m=+268.331324215" Nov 02 22:10:45.873825 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:45.873730 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cqbgl"] Nov 02 22:10:45.879884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:45.879861 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:45.910687 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:45.910646 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthbg\" (UniqueName: \"kubernetes.io/projected/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-kube-api-access-fthbg\") pod \"limitador-limitador-74dfcfb894-cqbgl\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:45.910687 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:45.910694 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-config-file\") pod \"limitador-limitador-74dfcfb894-cqbgl\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:45.930525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:45.930485 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 22:10:45.931813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:45.931781 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cqbgl"] Nov 02 22:10:46.011578 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:46.011525 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthbg\" (UniqueName: \"kubernetes.io/projected/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-kube-api-access-fthbg\") pod \"limitador-limitador-74dfcfb894-cqbgl\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:46.011774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:46.011608 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-config-file\") pod \"limitador-limitador-74dfcfb894-cqbgl\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:46.012343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:46.012319 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-config-file\") pod \"limitador-limitador-74dfcfb894-cqbgl\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:46.070072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:46.070036 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthbg\" (UniqueName: \"kubernetes.io/projected/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-kube-api-access-fthbg\") pod \"limitador-limitador-74dfcfb894-cqbgl\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:46.112993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:46.112944 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cqbgl"] Nov 02 22:10:46.193532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:46.193434 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:46.605693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:46.605660 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cqbgl"] Nov 02 22:10:46.609050 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:10:46.609008 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d62fea9_2e89_48ab_a1e6_2a6c05975f7c.slice/crio-d7c4b3a63d6de6a52363694f3d1f059697d081aaff5186c960bcd879c7b09d7f WatchSource:0}: Error finding container d7c4b3a63d6de6a52363694f3d1f059697d081aaff5186c960bcd879c7b09d7f: Status 404 returned error can't find the container with id d7c4b3a63d6de6a52363694f3d1f059697d081aaff5186c960bcd879c7b09d7f Nov 02 22:10:47.020134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.020095 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" event={"ID":"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c","Type":"ContainerStarted","Data":"d7c4b3a63d6de6a52363694f3d1f059697d081aaff5186c960bcd879c7b09d7f"} Nov 02 22:10:47.097349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.097311 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv"] Nov 02 22:10:47.100405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.100383 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.121802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.121776 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e241a34-aa8f-449b-9aa4-16a9d589ee82-metrics-certs\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.121941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.121817 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e241a34-aa8f-449b-9aa4-16a9d589ee82-cert\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.121941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.121840 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8e241a34-aa8f-449b-9aa4-16a9d589ee82-manager-config\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.122038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.121998 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vqw\" (UniqueName: \"kubernetes.io/projected/8e241a34-aa8f-449b-9aa4-16a9d589ee82-kube-api-access-r8vqw\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.150023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.149989 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 22:10:47.150023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.149988 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:10:47.196674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.196629 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 22:10:47.196674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.196636 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 22:10:47.196903 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.196648 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 22:10:47.196903 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.196710 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-7vzml\"" Nov 02 22:10:47.197735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.197718 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv"] Nov 02 22:10:47.223249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.223220 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vqw\" (UniqueName: \"kubernetes.io/projected/8e241a34-aa8f-449b-9aa4-16a9d589ee82-kube-api-access-r8vqw\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.223416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.223273 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e241a34-aa8f-449b-9aa4-16a9d589ee82-metrics-certs\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.223416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.223311 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e241a34-aa8f-449b-9aa4-16a9d589ee82-cert\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.223416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.223344 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8e241a34-aa8f-449b-9aa4-16a9d589ee82-manager-config\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.224134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.224114 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8e241a34-aa8f-449b-9aa4-16a9d589ee82-manager-config\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.226086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.226068 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e241a34-aa8f-449b-9aa4-16a9d589ee82-cert\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.226178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.226155 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e241a34-aa8f-449b-9aa4-16a9d589ee82-metrics-certs\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.291270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.291193 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vqw\" (UniqueName: \"kubernetes.io/projected/8e241a34-aa8f-449b-9aa4-16a9d589ee82-kube-api-access-r8vqw\") pod \"jobset-controller-manager-7655f9b8cd-8qmzv\" (UID: \"8e241a34-aa8f-449b-9aa4-16a9d589ee82\") " pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.412085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.412051 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:47.777165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:47.777128 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv"] Nov 02 22:10:47.780469 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:10:47.780436 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e241a34_aa8f_449b_9aa4_16a9d589ee82.slice/crio-382962aaa391c2b5231ceb552e7c6eb35f55707bd6663471648802c6a43785ba WatchSource:0}: Error finding container 382962aaa391c2b5231ceb552e7c6eb35f55707bd6663471648802c6a43785ba: Status 404 returned error can't find the container with id 382962aaa391c2b5231ceb552e7c6eb35f55707bd6663471648802c6a43785ba Nov 02 22:10:48.025645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:48.025606 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" event={"ID":"8e241a34-aa8f-449b-9aa4-16a9d589ee82","Type":"ContainerStarted","Data":"382962aaa391c2b5231ceb552e7c6eb35f55707bd6663471648802c6a43785ba"} Nov 02 22:10:51.041225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:51.041189 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" event={"ID":"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c","Type":"ContainerStarted","Data":"165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2"} Nov 02 22:10:51.041684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:51.041282 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:10:51.150603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:51.150522 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" podStartSLOduration=2.103356513 podStartE2EDuration="6.150501283s" totalImagesPullingTime="4.04714477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:51.149973832 +0000 UTC m=+287.414967052" watchObservedRunningTime="2026-11-02 22:10:51.150501283 +0000 UTC m=+287.415494505" Nov 02 22:10:54.054626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:54.054548 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" event={"ID":"8e241a34-aa8f-449b-9aa4-16a9d589ee82","Type":"ContainerStarted","Data":"0b92e43f3b2bea92b8367410b3707c6df5d90a02cb8a5ad79355d2c8b2a1e348"} Nov 02 22:10:54.055115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:54.054683 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:10:54.164548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:10:54.164485 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" podStartSLOduration=1.937000135 podStartE2EDuration="7.164468779s" totalImagesPullingTime="5.227468644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:54.163669291 +0000 UTC m=+290.428662513" watchObservedRunningTime="2026-11-02 22:10:54.164468779 +0000 UTC m=+290.429461998" Nov 02 22:10:56.489397 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:10:56.489363 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:11:02.047656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:02.047618 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:11:04.096850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:04.096809 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cqbgl"] Nov 02 22:11:04.097354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:04.097184 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" podUID="4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" containerName="limitador" containerID="cri-o://165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2" gracePeriod=30 Nov 02 22:11:05.063160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.063128 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7655f9b8cd-8qmzv" Nov 02 22:11:05.081892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.081853 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:11:05.101559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.101523 5109 generic.go:356] "Generic (PLEG): container finished" podID="4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" containerID="165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2" exitCode=0 Nov 02 22:11:05.101996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.101588 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" Nov 02 22:11:05.101996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.101614 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" event={"ID":"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c","Type":"ContainerDied","Data":"165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2"} Nov 02 22:11:05.101996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.101675 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cqbgl" event={"ID":"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c","Type":"ContainerDied","Data":"d7c4b3a63d6de6a52363694f3d1f059697d081aaff5186c960bcd879c7b09d7f"} Nov 02 22:11:05.101996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.101709 5109 scope.go:122] "RemoveContainer" containerID="165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2" Nov 02 22:11:05.110676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.110560 5109 scope.go:122] "RemoveContainer" containerID="165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2" Nov 02 22:11:05.110996 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:11:05.110971 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2\": container with ID starting with 165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2 not found: ID does not exist" containerID="165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2" Nov 02 22:11:05.111096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.111003 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2"} err="failed to get container status \"165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2\": rpc error: code = NotFound desc = could not find container \"165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2\": container with ID starting with 165f3c40a3a2cfc8e969f3f62de5a2fdb4f51452e291ced79e9c05335b5df8a2 not found: ID does not exist" Nov 02 22:11:05.185334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.185255 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-config-file\" (UniqueName: \"kubernetes.io/configmap/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-config-file\") pod \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " Nov 02 22:11:05.185490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.185406 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-kube-api-access-fthbg\" (UniqueName: \"kubernetes.io/projected/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-kube-api-access-fthbg\") pod \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\" (UID: \"4d62fea9-2e89-48ab-a1e6-2a6c05975f7c\") " Nov 02 22:11:05.186016 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.185697 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-config-file" pod "4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" (UID: "4d62fea9-2e89-48ab-a1e6-2a6c05975f7c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:05.186016 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.185826 5109 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-config-file\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:05.187951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.187918 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-kube-api-access-fthbg" pod "4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" (UID: "4d62fea9-2e89-48ab-a1e6-2a6c05975f7c"). InnerVolumeSpecName "kube-api-access-fthbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:05.287080 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.287040 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthbg\" (UniqueName: \"kubernetes.io/projected/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c-kube-api-access-fthbg\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:05.517858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.517820 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cqbgl"] Nov 02 22:11:05.522121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.522091 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cqbgl"] Nov 02 22:11:05.889765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:05.889681 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" path="/var/lib/kubelet/pods/4d62fea9-2e89-48ab-a1e6-2a6c05975f7c/volumes" Nov 02 22:11:08.248266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.248236 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg"] Nov 02 22:11:08.248765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.248712 5109 cpu_manager.go:414] "containerMap: removing container" podUID="4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" containerName="limitador" Nov 02 22:11:08.248765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.248730 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" containerName="limitador" Nov 02 22:11:08.248899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.248834 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d62fea9-2e89-48ab-a1e6-2a6c05975f7c" containerName="limitador" Nov 02 22:11:08.253261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.253240 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.302418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.302382 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:11:08.313896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.313865 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755vp\" (UniqueName: \"kubernetes.io/projected/43205cd0-50b8-4766-805b-25961e99fa03-kube-api-access-755vp\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.314077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.313907 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43205cd0-50b8-4766-805b-25961e99fa03-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.314077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.313948 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43205cd0-50b8-4766-805b-25961e99fa03-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.314170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.314065 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43205cd0-50b8-4766-805b-25961e99fa03-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.348984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.348957 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:11:08.349160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.349084 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-65bsc\"" Nov 02 22:11:08.414860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.414820 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755vp\" (UniqueName: \"kubernetes.io/projected/43205cd0-50b8-4766-805b-25961e99fa03-kube-api-access-755vp\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.415079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.414869 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43205cd0-50b8-4766-805b-25961e99fa03-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.415079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.414912 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43205cd0-50b8-4766-805b-25961e99fa03-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.415079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.414937 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43205cd0-50b8-4766-805b-25961e99fa03-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.415337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.415321 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43205cd0-50b8-4766-805b-25961e99fa03-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.415374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.415348 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43205cd0-50b8-4766-805b-25961e99fa03-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.443590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.443513 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 22:11:08.447737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.447697 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg"] Nov 02 22:11:08.449670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.449645 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43205cd0-50b8-4766-805b-25961e99fa03-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.475828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.475794 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755vp\" (UniqueName: \"kubernetes.io/projected/43205cd0-50b8-4766-805b-25961e99fa03-kube-api-access-755vp\") pod \"kserve-module-controller-manager-7566cf5fc8-p2rtg\" (UID: \"43205cd0-50b8-4766-805b-25961e99fa03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.565636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.565594 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:08.976466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.976441 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg"] Nov 02 22:11:08.979244 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:11:08.979212 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43205cd0_50b8_4766_805b_25961e99fa03.slice/crio-3a88eef8119999e0259a532e0f8f37e3b8906b521d1fe874ace1b670d92eb3db WatchSource:0}: Error finding container 3a88eef8119999e0259a532e0f8f37e3b8906b521d1fe874ace1b670d92eb3db: Status 404 returned error can't find the container with id 3a88eef8119999e0259a532e0f8f37e3b8906b521d1fe874ace1b670d92eb3db Nov 02 22:11:08.981357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:08.981339 5109 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:11:09.120457 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.120417 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" event={"ID":"43205cd0-50b8-4766-805b-25961e99fa03","Type":"ContainerStarted","Data":"3a88eef8119999e0259a532e0f8f37e3b8906b521d1fe874ace1b670d92eb3db"} Nov 02 22:11:09.785738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.785704 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh"] Nov 02 22:11:09.788734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.788715 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:09.838427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.838396 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:11:09.838728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.838710 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:11:09.838807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.838721 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 22:11:09.885144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.885104 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5679f\"" Nov 02 22:11:09.889539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.889517 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh"] Nov 02 22:11:09.929410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.929372 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69bc7953-cb2e-4743-98af-65a14cac0ec1-data\") pod \"postgres-7cdbf6dc44-4bckh\" (UID: \"69bc7953-cb2e-4743-98af-65a14cac0ec1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:09.929410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:09.929414 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5ghr\" (UniqueName: \"kubernetes.io/projected/69bc7953-cb2e-4743-98af-65a14cac0ec1-kube-api-access-h5ghr\") pod \"postgres-7cdbf6dc44-4bckh\" (UID: \"69bc7953-cb2e-4743-98af-65a14cac0ec1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:10.030783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:10.030748 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69bc7953-cb2e-4743-98af-65a14cac0ec1-data\") pod \"postgres-7cdbf6dc44-4bckh\" (UID: \"69bc7953-cb2e-4743-98af-65a14cac0ec1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:10.030960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:10.030787 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5ghr\" (UniqueName: \"kubernetes.io/projected/69bc7953-cb2e-4743-98af-65a14cac0ec1-kube-api-access-h5ghr\") pod \"postgres-7cdbf6dc44-4bckh\" (UID: \"69bc7953-cb2e-4743-98af-65a14cac0ec1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:10.031221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:10.031202 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69bc7953-cb2e-4743-98af-65a14cac0ec1-data\") pod \"postgres-7cdbf6dc44-4bckh\" (UID: \"69bc7953-cb2e-4743-98af-65a14cac0ec1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:10.090053 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:10.089986 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5ghr\" (UniqueName: \"kubernetes.io/projected/69bc7953-cb2e-4743-98af-65a14cac0ec1-kube-api-access-h5ghr\") pod \"postgres-7cdbf6dc44-4bckh\" (UID: \"69bc7953-cb2e-4743-98af-65a14cac0ec1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:10.100832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:10.100792 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:10.483430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:10.483246 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh"] Nov 02 22:11:10.486747 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:11:10.486714 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bc7953_cb2e_4743_98af_65a14cac0ec1.slice/crio-1e02ab6146e21a14eef8148ef6b5e75c18fc68cac97a08beef694c7c13375b63 WatchSource:0}: Error finding container 1e02ab6146e21a14eef8148ef6b5e75c18fc68cac97a08beef694c7c13375b63: Status 404 returned error can't find the container with id 1e02ab6146e21a14eef8148ef6b5e75c18fc68cac97a08beef694c7c13375b63 Nov 02 22:11:11.132535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:11.132487 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" event={"ID":"69bc7953-cb2e-4743-98af-65a14cac0ec1","Type":"ContainerStarted","Data":"1e02ab6146e21a14eef8148ef6b5e75c18fc68cac97a08beef694c7c13375b63"} Nov 02 22:11:12.139936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:12.139892 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" event={"ID":"43205cd0-50b8-4766-805b-25961e99fa03","Type":"ContainerStarted","Data":"aa9a91e7d1301b7ef41d5a5192c62e6f8b287039e69fc73222b51be98af43e07"} Nov 02 22:11:12.140392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:12.140036 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:12.268973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:12.268904 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" podStartSLOduration=1.200818554 podStartE2EDuration="4.268884598s" totalImagesPullingTime="3.068066044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:12.268473489 +0000 UTC m=+308.533466708" watchObservedRunningTime="2026-11-02 22:11:12.268884598 +0000 UTC m=+308.533877819" Nov 02 22:11:18.169154 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:18.169114 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" event={"ID":"69bc7953-cb2e-4743-98af-65a14cac0ec1","Type":"ContainerStarted","Data":"f423073ab36642b90abf91bb85ae67527764af0ea358bccf437fd8b9aa02971a"} Nov 02 22:11:18.169621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:18.169235 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:18.278513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:18.278424 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" podStartSLOduration=2.41922653 podStartE2EDuration="9.278405765s" totalImagesPullingTime="6.859179235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:18.277919592 +0000 UTC m=+314.542912811" watchObservedRunningTime="2026-11-02 22:11:18.278405765 +0000 UTC m=+314.543398985" Nov 02 22:11:24.204475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:24.204434 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4bckh" Nov 02 22:11:26.490004 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:11:26.489965 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:11:32.044130 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.044091 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-546bfbd88f-nd9wf"] Nov 02 22:11:32.055203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.055169 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.105804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.105770 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-nd9wf"] Nov 02 22:11:32.106086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.105858 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:11:32.141655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.141618 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-metrics-certs\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.141655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.141658 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmjp\" (UniqueName: \"kubernetes.io/projected/805d005f-e957-4d8b-a6d9-6978b429c1d6-kube-api-access-gbmjp\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.141869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.141781 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-webhook-cert\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.151109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.151073 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:11:32.151266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.151085 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:11:32.151266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.151127 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jsb8q\"" Nov 02 22:11:32.242963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.242925 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-metrics-certs\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.243150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.242969 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmjp\" (UniqueName: \"kubernetes.io/projected/805d005f-e957-4d8b-a6d9-6978b429c1d6-kube-api-access-gbmjp\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.243200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.243176 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-webhook-cert\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.245711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.245682 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-webhook-cert\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.245861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.245730 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-metrics-certs\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.301049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.300966 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmjp\" (UniqueName: \"kubernetes.io/projected/805d005f-e957-4d8b-a6d9-6978b429c1d6-kube-api-access-gbmjp\") pod \"maas-controller-546bfbd88f-nd9wf\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.367921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.367877 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:32.735403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:32.735312 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-nd9wf"] Nov 02 22:11:32.767591 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:11:32.767532 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805d005f_e957_4d8b_a6d9_6978b429c1d6.slice/crio-e5c1c0227cc074d821b55365b75baa8dade1b8905b499274b331ecfad646e65a WatchSource:0}: Error finding container e5c1c0227cc074d821b55365b75baa8dade1b8905b499274b331ecfad646e65a: Status 404 returned error can't find the container with id e5c1c0227cc074d821b55365b75baa8dade1b8905b499274b331ecfad646e65a Nov 02 22:11:33.146909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:33.146877 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p2rtg" Nov 02 22:11:33.235119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:33.235073 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" event={"ID":"805d005f-e957-4d8b-a6d9-6978b429c1d6","Type":"ContainerStarted","Data":"e5c1c0227cc074d821b55365b75baa8dade1b8905b499274b331ecfad646e65a"} Nov 02 22:11:36.249907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:36.249811 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" event={"ID":"805d005f-e957-4d8b-a6d9-6978b429c1d6","Type":"ContainerStarted","Data":"547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd"} Nov 02 22:11:36.250294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:36.249958 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:36.362465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:36.362406 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" podStartSLOduration=1.196216799 podStartE2EDuration="4.362387483s" totalImagesPullingTime="3.166170684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:36.36129786 +0000 UTC m=+332.626291079" watchObservedRunningTime="2026-11-02 22:11:36.362387483 +0000 UTC m=+332.627380702" Nov 02 22:11:47.257264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:11:47.257233 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:11:56.489792 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:11:56.489747 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:12:10.009323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.009281 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp"] Nov 02 22:12:10.044830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.044800 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.090349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.090302 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f20cd24-bbe4-417b-911d-937733fcf433-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.090604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.090361 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6k6\" (UniqueName: \"kubernetes.io/projected/9f20cd24-bbe4-417b-911d-937733fcf433-kube-api-access-mk6k6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.090604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.090529 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.090604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.090592 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.090766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.090632 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.090766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.090667 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f20cd24-bbe4-417b-911d-937733fcf433-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.090828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.090767 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f20cd24-bbe4-417b-911d-937733fcf433-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.146626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.146583 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp"] Nov 02 22:12:10.192014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.191975 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f20cd24-bbe4-417b-911d-937733fcf433-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.192014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.192030 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f20cd24-bbe4-417b-911d-937733fcf433-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.192323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.192066 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f20cd24-bbe4-417b-911d-937733fcf433-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.192323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.192090 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6k6\" (UniqueName: \"kubernetes.io/projected/9f20cd24-bbe4-417b-911d-937733fcf433-kube-api-access-mk6k6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.192323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.192126 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.192323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.192159 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.192323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.192195 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.192932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.192895 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f20cd24-bbe4-417b-911d-937733fcf433-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.195017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.194969 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f20cd24-bbe4-417b-911d-937733fcf433-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.195017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.195002 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.195179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.195029 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.195179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.195124 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f20cd24-bbe4-417b-911d-937733fcf433-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.250717 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.250681 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f20cd24-bbe4-417b-911d-937733fcf433-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.250870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.250831 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6k6\" (UniqueName: \"kubernetes.io/projected/9f20cd24-bbe4-417b-911d-937733fcf433-kube-api-access-mk6k6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mcdtp\" (UID: \"9f20cd24-bbe4-417b-911d-937733fcf433\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.358090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.357989 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:10.779325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.779232 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp"] Nov 02 22:12:10.783335 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:12:10.782228 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f20cd24_bbe4_417b_911d_937733fcf433.slice/crio-4bc1329a86e26961175a17330f4677fc0c8e9b257860db4b311668379f111aa2 WatchSource:0}: Error finding container 4bc1329a86e26961175a17330f4677fc0c8e9b257860db4b311668379f111aa2: Status 404 returned error can't find the container with id 4bc1329a86e26961175a17330f4677fc0c8e9b257860db4b311668379f111aa2 Nov 02 22:12:10.785944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.785908 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:12:10.786047 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:10.785978 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:12:11.407684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:11.407592 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" event={"ID":"9f20cd24-bbe4-417b-911d-937733fcf433","Type":"ContainerStarted","Data":"cc9ff88ff48aa16ac58daf2c4a1877bbf6b7ae1132401e5d2d569b48aaf6c0fc"} Nov 02 22:12:11.407684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:11.407655 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" event={"ID":"9f20cd24-bbe4-417b-911d-937733fcf433","Type":"ContainerStarted","Data":"4bc1329a86e26961175a17330f4677fc0c8e9b257860db4b311668379f111aa2"} Nov 02 22:12:11.408153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:11.407777 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:11.409616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:11.409594 5109 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mcdtp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:11.409730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:11.409638 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" podUID="9f20cd24-bbe4-417b-911d-937733fcf433" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:11.602329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:11.602272 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" podStartSLOduration=2.602258682 podStartE2EDuration="2.602258682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:11.568627031 +0000 UTC m=+367.833620251" watchObservedRunningTime="2026-11-02 22:12:11.602258682 +0000 UTC m=+367.867251900" Nov 02 22:12:12.412615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.412560 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mcdtp" Nov 02 22:12:12.574303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.574235 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz"] Nov 02 22:12:12.574533 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.574482 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" podUID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" containerName="discovery" containerID="cri-o://b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c" gracePeriod=30 Nov 02 22:12:12.595757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.595711 5109 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-l24hz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.17:8080/ready\": dial tcp 10.133.0.17:8080: connect: connection refused" start-of-body= Nov 02 22:12:12.595905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.595787 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" podUID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.17:8080/ready\": dial tcp 10.133.0.17:8080: connect: connection refused" Nov 02 22:12:12.825918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.825886 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:12:12.916786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.916742 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-cacerts\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-cacerts\") pod \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " Nov 02 22:12:12.916974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.916821 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-token\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-token\") pod \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " Nov 02 22:12:12.916974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.916842 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-kubeconfig\") pod \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " Nov 02 22:12:12.916974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.916861 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68ca43d-8fcb-4815-af29-a4ed75f36c75-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68ca43d-8fcb-4815-af29-a4ed75f36c75-local-certs\") pod \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " Nov 02 22:12:12.916974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.916878 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-dns-cert\") pod \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " Nov 02 22:12:12.916974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.916928 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-ca-configmap\") pod \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " Nov 02 22:12:12.916974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.916960 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-kube-api-access-grc57\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-kube-api-access-grc57\") pod \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\" (UID: \"f68ca43d-8fcb-4815-af29-a4ed75f36c75\") " Nov 02 22:12:12.917902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.917371 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-ca-configmap" pod "f68ca43d-8fcb-4815-af29-a4ed75f36c75" (UID: "f68ca43d-8fcb-4815-af29-a4ed75f36c75"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:12.919515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.919489 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-kubeconfig" pod "f68ca43d-8fcb-4815-af29-a4ed75f36c75" (UID: "f68ca43d-8fcb-4815-af29-a4ed75f36c75"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:12.919745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.919635 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68ca43d-8fcb-4815-af29-a4ed75f36c75-local-certs" pod "f68ca43d-8fcb-4815-af29-a4ed75f36c75" (UID: "f68ca43d-8fcb-4815-af29-a4ed75f36c75"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:12.919745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.919663 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-kube-api-access-grc57" pod "f68ca43d-8fcb-4815-af29-a4ed75f36c75" (UID: "f68ca43d-8fcb-4815-af29-a4ed75f36c75"). InnerVolumeSpecName "kube-api-access-grc57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:12.919745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.919674 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-cacerts" pod "f68ca43d-8fcb-4815-af29-a4ed75f36c75" (UID: "f68ca43d-8fcb-4815-af29-a4ed75f36c75"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:12.919745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.919717 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-token" pod "f68ca43d-8fcb-4815-af29-a4ed75f36c75" (UID: "f68ca43d-8fcb-4815-af29-a4ed75f36c75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:12.919894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:12.919792 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-dns-cert" pod "f68ca43d-8fcb-4815-af29-a4ed75f36c75" (UID: "f68ca43d-8fcb-4815-af29-a4ed75f36c75"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:13.018359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.018319 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grc57\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-kube-api-access-grc57\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:13.018359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.018352 5109 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-cacerts\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:13.018359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.018362 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:13.018689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.018372 5109 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-kubeconfig\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:13.018689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.018381 5109 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68ca43d-8fcb-4815-af29-a4ed75f36c75-local-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:13.018689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.018389 5109 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-dns-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:13.018689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.018399 5109 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68ca43d-8fcb-4815-af29-a4ed75f36c75-istio-csr-ca-configmap\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:13.417138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.417098 5109 generic.go:356] "Generic (PLEG): container finished" podID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" containerID="b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c" exitCode=0 Nov 02 22:12:13.417585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.417142 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" Nov 02 22:12:13.417585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.417195 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" event={"ID":"f68ca43d-8fcb-4815-af29-a4ed75f36c75","Type":"ContainerDied","Data":"b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c"} Nov 02 22:12:13.417585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.417245 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz" event={"ID":"f68ca43d-8fcb-4815-af29-a4ed75f36c75","Type":"ContainerDied","Data":"978c66146d53b31ad494991d3f60949038a3b515790c2bcbb5bd7e0a459afb03"} Nov 02 22:12:13.417585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.417271 5109 scope.go:122] "RemoveContainer" containerID="b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c" Nov 02 22:12:13.428467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.428447 5109 scope.go:122] "RemoveContainer" containerID="b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c" Nov 02 22:12:13.428807 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:12:13.428786 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c\": container with ID starting with b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c not found: ID does not exist" containerID="b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c" Nov 02 22:12:13.428873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.428820 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c"} err="failed to get container status \"b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c\": rpc error: code = NotFound desc = could not find container \"b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c\": container with ID starting with b2eb1b056c912432df659e47fe7f847f8cdaf78b73e416b3c053fe8169a73f3c not found: ID does not exist" Nov 02 22:12:13.572351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.572314 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz"] Nov 02 22:12:13.619189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.619147 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l24hz"] Nov 02 22:12:13.891678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:13.891634 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" path="/var/lib/kubelet/pods/f68ca43d-8fcb-4815-af29-a4ed75f36c75/volumes" Nov 02 22:12:15.184394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.184355 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hbxxd"] Nov 02 22:12:15.184812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.184731 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" containerName="discovery" Nov 02 22:12:15.184812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.184742 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" containerName="discovery" Nov 02 22:12:15.184886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.184821 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68ca43d-8fcb-4815-af29-a4ed75f36c75" containerName="discovery" Nov 02 22:12:15.189222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.189200 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.238770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.238733 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e10c6a1-006b-42ce-b75e-be8f3ce0031d-cert\") pod \"kserve-controller-manager-776cc66d47-hbxxd\" (UID: \"0e10c6a1-006b-42ce-b75e-be8f3ce0031d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.238923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.238781 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdwp\" (UniqueName: \"kubernetes.io/projected/0e10c6a1-006b-42ce-b75e-be8f3ce0031d-kube-api-access-hgdwp\") pod \"kserve-controller-manager-776cc66d47-hbxxd\" (UID: \"0e10c6a1-006b-42ce-b75e-be8f3ce0031d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.239105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.239087 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jtqts\"" Nov 02 22:12:15.239142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.239091 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 22:12:15.252296 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.252243 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hbxxd"] Nov 02 22:12:15.339895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.339854 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e10c6a1-006b-42ce-b75e-be8f3ce0031d-cert\") pod \"kserve-controller-manager-776cc66d47-hbxxd\" (UID: \"0e10c6a1-006b-42ce-b75e-be8f3ce0031d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.340088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.339905 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdwp\" (UniqueName: \"kubernetes.io/projected/0e10c6a1-006b-42ce-b75e-be8f3ce0031d-kube-api-access-hgdwp\") pod \"kserve-controller-manager-776cc66d47-hbxxd\" (UID: \"0e10c6a1-006b-42ce-b75e-be8f3ce0031d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.342606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.342582 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e10c6a1-006b-42ce-b75e-be8f3ce0031d-cert\") pod \"kserve-controller-manager-776cc66d47-hbxxd\" (UID: \"0e10c6a1-006b-42ce-b75e-be8f3ce0031d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.398306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.398269 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdwp\" (UniqueName: \"kubernetes.io/projected/0e10c6a1-006b-42ce-b75e-be8f3ce0031d-kube-api-access-hgdwp\") pod \"kserve-controller-manager-776cc66d47-hbxxd\" (UID: \"0e10c6a1-006b-42ce-b75e-be8f3ce0031d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.407801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.406373 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8tqw4"] Nov 02 22:12:15.409986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.409945 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.440856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.440758 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3308a94-eb6a-47eb-9283-be6bfd598b5f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8tqw4\" (UID: \"e3308a94-eb6a-47eb-9283-be6bfd598b5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.440856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.440804 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvdk\" (UniqueName: \"kubernetes.io/projected/e3308a94-eb6a-47eb-9283-be6bfd598b5f-kube-api-access-jfvdk\") pod \"model-serving-api-75bd89fdfc-8tqw4\" (UID: \"e3308a94-eb6a-47eb-9283-be6bfd598b5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.459052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.459018 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fmr7m\"" Nov 02 22:12:15.459052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.459044 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 22:12:15.465618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.465580 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8tqw4"] Nov 02 22:12:15.503451 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.503415 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:15.541540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.541491 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3308a94-eb6a-47eb-9283-be6bfd598b5f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8tqw4\" (UID: \"e3308a94-eb6a-47eb-9283-be6bfd598b5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.541540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.541543 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvdk\" (UniqueName: \"kubernetes.io/projected/e3308a94-eb6a-47eb-9283-be6bfd598b5f-kube-api-access-jfvdk\") pod \"model-serving-api-75bd89fdfc-8tqw4\" (UID: \"e3308a94-eb6a-47eb-9283-be6bfd598b5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.544580 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.544535 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3308a94-eb6a-47eb-9283-be6bfd598b5f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8tqw4\" (UID: \"e3308a94-eb6a-47eb-9283-be6bfd598b5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.602135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.602099 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvdk\" (UniqueName: \"kubernetes.io/projected/e3308a94-eb6a-47eb-9283-be6bfd598b5f-kube-api-access-jfvdk\") pod \"model-serving-api-75bd89fdfc-8tqw4\" (UID: \"e3308a94-eb6a-47eb-9283-be6bfd598b5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.724427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.724320 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:15.874378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:15.874162 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hbxxd"] Nov 02 22:12:15.876788 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:12:15.876759 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e10c6a1_006b_42ce_b75e_be8f3ce0031d.slice/crio-8095026df68b0ce804ded5e8759ef7ec811180d9ecf58b7c948cb5c8560cbad5 WatchSource:0}: Error finding container 8095026df68b0ce804ded5e8759ef7ec811180d9ecf58b7c948cb5c8560cbad5: Status 404 returned error can't find the container with id 8095026df68b0ce804ded5e8759ef7ec811180d9ecf58b7c948cb5c8560cbad5 Nov 02 22:12:16.091703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:16.091671 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8tqw4"] Nov 02 22:12:16.094248 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:12:16.094207 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3308a94_eb6a_47eb_9283_be6bfd598b5f.slice/crio-b7292d2628db6514504a6acf48bfc3b4e6597a79ed5ec39a4eb69e7865aa51cf WatchSource:0}: Error finding container b7292d2628db6514504a6acf48bfc3b4e6597a79ed5ec39a4eb69e7865aa51cf: Status 404 returned error can't find the container with id b7292d2628db6514504a6acf48bfc3b4e6597a79ed5ec39a4eb69e7865aa51cf Nov 02 22:12:16.433686 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:16.433592 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" event={"ID":"0e10c6a1-006b-42ce-b75e-be8f3ce0031d","Type":"ContainerStarted","Data":"8095026df68b0ce804ded5e8759ef7ec811180d9ecf58b7c948cb5c8560cbad5"} Nov 02 22:12:16.434792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:16.434760 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" event={"ID":"e3308a94-eb6a-47eb-9283-be6bfd598b5f","Type":"ContainerStarted","Data":"b7292d2628db6514504a6acf48bfc3b4e6597a79ed5ec39a4eb69e7865aa51cf"} Nov 02 22:12:19.452631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:19.452588 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" event={"ID":"e3308a94-eb6a-47eb-9283-be6bfd598b5f","Type":"ContainerStarted","Data":"b0e323dd82779f4ce5d56d264bf57508e4d67cc6c7365fbe78c2e53aa68eee0d"} Nov 02 22:12:19.453153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:19.452775 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:19.454226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:19.454198 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" event={"ID":"0e10c6a1-006b-42ce-b75e-be8f3ce0031d","Type":"ContainerStarted","Data":"a82e4583be645af77dcd3bbb4b9dd82eff0570762062efee8505b28ef286db23"} Nov 02 22:12:19.454346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:19.454283 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:19.573117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:19.573054 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" podStartSLOduration=2.01257131 podStartE2EDuration="4.573034982s" totalImagesPullingTime="2.560463672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:19.572711569 +0000 UTC m=+375.837704788" watchObservedRunningTime="2026-11-02 22:12:19.573034982 +0000 UTC m=+375.838028201" Nov 02 22:12:19.685372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:19.685318 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" podStartSLOduration=1.915978495 podStartE2EDuration="4.685298889s" totalImagesPullingTime="2.769320394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:19.685181208 +0000 UTC m=+375.950174429" watchObservedRunningTime="2026-11-02 22:12:19.685298889 +0000 UTC m=+375.950292105" Nov 02 22:12:26.492260 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:12:26.492200 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:12:30.463061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:30.463022 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-8tqw4" Nov 02 22:12:50.464265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:12:50.464213 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hbxxd" Nov 02 22:12:56.489464 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:12:56.489423 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:13:26.489529 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:13:26.489489 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:13:56.490608 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:13:56.490547 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:14:24.170311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.170267 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8"] Nov 02 22:14:24.173907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.173882 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.223504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.223474 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:14:24.242192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.242157 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.242356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.242212 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.242356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.242290 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqh75\" (UniqueName: \"kubernetes.io/projected/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-kube-api-access-jqh75\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.242356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.242334 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.242356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.242354 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.242486 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.242372 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.269911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.269879 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:14:24.270091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.270068 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-swvkh\"" Nov 02 22:14:24.270131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.270092 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 22:14:24.271329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.271309 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8"] Nov 02 22:14:24.342912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.342878 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.342924 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.342947 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqh75\" (UniqueName: \"kubernetes.io/projected/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-kube-api-access-jqh75\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.342982 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.343001 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.343016 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.343443 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.343469 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.343870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.343488 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.345487 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.345465 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.345601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.345549 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.401732 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.401699 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqh75\" (UniqueName: \"kubernetes.io/projected/2f8e0bdb-a3c5-45b7-8539-4454e9da8d23-kube-api-access-jqh75\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8\" (UID: \"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.489465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.489380 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:24.862371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:24.862346 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8"] Nov 02 22:14:24.864980 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:14:24.864942 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8e0bdb_a3c5_45b7_8539_4454e9da8d23.slice/crio-58164af8265aa9d9e1341b0350602dbb4a96752313074d123f5c13467a85bff9 WatchSource:0}: Error finding container 58164af8265aa9d9e1341b0350602dbb4a96752313074d123f5c13467a85bff9: Status 404 returned error can't find the container with id 58164af8265aa9d9e1341b0350602dbb4a96752313074d123f5c13467a85bff9 Nov 02 22:14:25.012453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:25.012414 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" event={"ID":"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23","Type":"ContainerStarted","Data":"58164af8265aa9d9e1341b0350602dbb4a96752313074d123f5c13467a85bff9"} Nov 02 22:14:26.491726 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:14:26.491683 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:14:28.027617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:28.027562 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" event={"ID":"2f8e0bdb-a3c5-45b7-8539-4454e9da8d23","Type":"ContainerStarted","Data":"bf59f4ab1b2d9144b09ac803da4773d15c447b570512c6cbca80c17d822268e6"} Nov 02 22:14:28.140159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:28.140094 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" podStartSLOduration=1.9980715980000001 podStartE2EDuration="4.140079993s" totalImagesPullingTime="2.142008395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:28.139450961 +0000 UTC m=+504.404444181" watchObservedRunningTime="2026-11-02 22:14:28.140079993 +0000 UTC m=+504.405073212" Nov 02 22:14:28.489743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:28.489643 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:32.804338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.804298 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9"] Nov 02 22:14:32.807794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.807771 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:32.903509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.903476 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 22:14:32.904816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.904794 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9"] Nov 02 22:14:32.917270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.917231 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edb523e3-3fad-4570-9d85-3c7798aba687-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:32.917270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.917276 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:32.917470 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.917335 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:32.917470 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.917357 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:32.917554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.917497 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:32.917627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:32.917544 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srd9\" (UniqueName: \"kubernetes.io/projected/edb523e3-3fad-4570-9d85-3c7798aba687-kube-api-access-2srd9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.018911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.018868 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.018911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.018913 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srd9\" (UniqueName: \"kubernetes.io/projected/edb523e3-3fad-4570-9d85-3c7798aba687-kube-api-access-2srd9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.019247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.018954 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edb523e3-3fad-4570-9d85-3c7798aba687-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.022205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.019711 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.022205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.019839 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.022205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.019904 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.022205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.020208 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.022205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.020453 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.022631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.022428 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.022881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.022858 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edb523e3-3fad-4570-9d85-3c7798aba687-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.027208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.027181 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edb523e3-3fad-4570-9d85-3c7798aba687-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.078070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.077994 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srd9\" (UniqueName: \"kubernetes.io/projected/edb523e3-3fad-4570-9d85-3c7798aba687-kube-api-access-2srd9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9\" (UID: \"edb523e3-3fad-4570-9d85-3c7798aba687\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.120126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.120086 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:33.489237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:33.489211 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9"] Nov 02 22:14:33.492033 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:14:33.491990 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb523e3_3fad_4570_9d85_3c7798aba687.slice/crio-6e60ecc620fa72586d836a8933c34aebc462207bf2d593d7eed14262c66d8e6d WatchSource:0}: Error finding container 6e60ecc620fa72586d836a8933c34aebc462207bf2d593d7eed14262c66d8e6d: Status 404 returned error can't find the container with id 6e60ecc620fa72586d836a8933c34aebc462207bf2d593d7eed14262c66d8e6d Nov 02 22:14:34.054513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:34.054476 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" event={"ID":"edb523e3-3fad-4570-9d85-3c7798aba687","Type":"ContainerStarted","Data":"6e60ecc620fa72586d836a8933c34aebc462207bf2d593d7eed14262c66d8e6d"} Nov 02 22:14:34.490317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:34.490217 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:34.503175 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:34.503137 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:35.060530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:35.060494 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" event={"ID":"edb523e3-3fad-4570-9d85-3c7798aba687","Type":"ContainerStarted","Data":"9ae2878a65b80655a4c95fca1065a5bb9d4bbf09913918e1d49ea41cbcca7a35"} Nov 02 22:14:35.072447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:35.072419 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8" Nov 02 22:14:35.120837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:35.120790 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:35.171610 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:35.171527 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" podStartSLOduration=2.544075918 podStartE2EDuration="3.171509209s" totalImagesPullingTime="627.433291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:35.170933019 +0000 UTC m=+511.435926253" watchObservedRunningTime="2026-11-02 22:14:35.171509209 +0000 UTC m=+511.436502429" Nov 02 22:14:39.385701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.385665 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c"] Nov 02 22:14:39.389347 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.389324 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.439470 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.439431 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q825d\"" Nov 02 22:14:39.476063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476011 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476080 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8cv\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-kube-api-access-ks8cv\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476208 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476255 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476351 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476392 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476414 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476604 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.476690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.476654 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.488537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.488503 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c"] Nov 02 22:14:39.577782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.577744 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.577782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.577779 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.577810 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.577831 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.577951 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.577997 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8cv\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-kube-api-access-ks8cv\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578050 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578074 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578149 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578238 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578291 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578470 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578520 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.578787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.578762 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.580404 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.580380 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.580404 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.580397 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.636864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.636779 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.637017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.636923 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8cv\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-kube-api-access-ks8cv\") pod \"maas-default-gateway-openshift-default-76f4876667-hfn4c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:39.702910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:39.702862 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:40.119347 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:40.119322 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c"] Nov 02 22:14:40.121722 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:14:40.121692 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d00dd36_6f5d_42a9_b59d_1acdb1db6a9c.slice/crio-f324670010580818cdcf353e96880a36a604af34d5c7def2a5952ae0014cacea WatchSource:0}: Error finding container f324670010580818cdcf353e96880a36a604af34d5c7def2a5952ae0014cacea: Status 404 returned error can't find the container with id f324670010580818cdcf353e96880a36a604af34d5c7def2a5952ae0014cacea Nov 02 22:14:41.091953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:41.091913 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" event={"ID":"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c","Type":"ContainerStarted","Data":"f324670010580818cdcf353e96880a36a604af34d5c7def2a5952ae0014cacea"} Nov 02 22:14:43.120284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:43.120247 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:43.137751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:43.137726 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:44.120741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:44.120707 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9" Nov 02 22:14:44.488717 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:44.488689 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:14:44.488797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:44.488756 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:14:44.488797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:44.488786 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:14:45.112226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.112180 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" event={"ID":"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c","Type":"ContainerStarted","Data":"c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1"} Nov 02 22:14:45.274882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.274823 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podStartSLOduration=1.910732809 podStartE2EDuration="6.274808892s" totalImagesPullingTime="4.364076083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:45.273720802 +0000 UTC m=+521.538714021" watchObservedRunningTime="2026-11-02 22:14:45.274808892 +0000 UTC m=+521.539802110" Nov 02 22:14:45.563289 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.563252 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9"] Nov 02 22:14:45.567099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.567077 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.616692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.616662 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 22:14:45.638606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.624550 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9"] Nov 02 22:14:45.653153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.653111 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.653344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.653193 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.653344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.653229 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.653344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.653280 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207b3952-7783-4159-ba11-835f76be56f4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.653344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.653312 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9nq\" (UniqueName: \"kubernetes.io/projected/207b3952-7783-4159-ba11-835f76be56f4-kube-api-access-xh9nq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.653344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.653342 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.704104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.704064 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:45.705464 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.705430 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-hfn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 22:14:45.705632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.705487 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 22:14:45.753899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.753867 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.753899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.753901 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.754120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.753940 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207b3952-7783-4159-ba11-835f76be56f4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.754120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.753960 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9nq\" (UniqueName: \"kubernetes.io/projected/207b3952-7783-4159-ba11-835f76be56f4-kube-api-access-xh9nq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.754120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.753979 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.754276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.754211 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.754319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.754301 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.754361 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.754319 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.754548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.754529 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.756435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.756415 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207b3952-7783-4159-ba11-835f76be56f4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.756756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.756735 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207b3952-7783-4159-ba11-835f76be56f4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.816121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.816040 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9nq\" (UniqueName: \"kubernetes.io/projected/207b3952-7783-4159-ba11-835f76be56f4-kube-api-access-xh9nq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8hsg9\" (UID: \"207b3952-7783-4159-ba11-835f76be56f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:45.880246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:45.880197 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:46.277115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:46.277088 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9"] Nov 02 22:14:46.279819 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:14:46.279783 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207b3952_7783_4159_ba11_835f76be56f4.slice/crio-e928f99e2c90f17c8e46ed93fd44d3cdeae5f0d3f5ff77afd85c5dd21e333598 WatchSource:0}: Error finding container e928f99e2c90f17c8e46ed93fd44d3cdeae5f0d3f5ff77afd85c5dd21e333598: Status 404 returned error can't find the container with id e928f99e2c90f17c8e46ed93fd44d3cdeae5f0d3f5ff77afd85c5dd21e333598 Nov 02 22:14:46.703513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:46.703427 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-hfn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 22:14:46.703513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:46.703481 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 22:14:47.123011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:47.122974 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" event={"ID":"207b3952-7783-4159-ba11-835f76be56f4","Type":"ContainerStarted","Data":"79c6d321347d7ed145a34a3957747c5fa5f2b90528116a0ef0d1542c6b0329cd"} Nov 02 22:14:47.123203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:47.123032 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" event={"ID":"207b3952-7783-4159-ba11-835f76be56f4","Type":"ContainerStarted","Data":"e928f99e2c90f17c8e46ed93fd44d3cdeae5f0d3f5ff77afd85c5dd21e333598"} Nov 02 22:14:47.703805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:47.703766 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-hfn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 22:14:47.704334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:47.703826 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 22:14:47.881246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:47.881204 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:48.703250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:48.703208 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-hfn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 22:14:48.703402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:48.703262 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 22:14:49.704264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:49.704225 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-hfn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 22:14:49.704702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:49.704291 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 22:14:50.703930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:50.703894 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-hfn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 22:14:50.704108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:50.703965 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 22:14:52.480782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.480727 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" podStartSLOduration=6.853785975 podStartE2EDuration="7.480711127s" totalImagesPullingTime="626.925152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:47.240535007 +0000 UTC m=+523.505528227" watchObservedRunningTime="2026-11-02 22:14:52.480711127 +0000 UTC m=+528.745704347" Nov 02 22:14:52.481755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.481732 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts"] Nov 02 22:14:52.485216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.485196 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.534814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.534786 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 22:14:52.582587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.582536 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts"] Nov 02 22:14:52.618346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.618306 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.618346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.618350 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgwd\" (UniqueName: \"kubernetes.io/projected/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-kube-api-access-4sgwd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.618594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.618383 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.618594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.618503 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.618673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.618618 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.618673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.618638 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.703731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.703685 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-hfn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:52.703936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.703795 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:52.719615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.719588 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.719738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.719638 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.719738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.719671 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.719738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.719693 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgwd\" (UniqueName: \"kubernetes.io/projected/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-kube-api-access-4sgwd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.719738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.719720 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.719934 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.719887 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.720045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.720017 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.720241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.720224 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.720312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.720287 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.722026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.722007 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.722395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.722375 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.778582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.778528 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgwd\" (UniqueName: \"kubernetes.io/projected/f2124f9b-a6c2-401d-99fd-8c0ecf601f26-kube-api-access-4sgwd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts\" (UID: \"f2124f9b-a6c2-401d-99fd-8c0ecf601f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:52.798067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:52.798036 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:53.180809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:53.180721 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts"] Nov 02 22:14:53.186755 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:14:53.186704 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2124f9b_a6c2_401d_99fd_8c0ecf601f26.slice/crio-afe14ab6092d795ad487f35730f2a870685343d27096e7203e5a93edd859581c WatchSource:0}: Error finding container afe14ab6092d795ad487f35730f2a870685343d27096e7203e5a93edd859581c: Status 404 returned error can't find the container with id afe14ab6092d795ad487f35730f2a870685343d27096e7203e5a93edd859581c Nov 02 22:14:53.678466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:53.678432 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:53.679159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:53.679133 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:53.724401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:53.724373 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:14:54.191201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:54.191162 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" event={"ID":"f2124f9b-a6c2-401d-99fd-8c0ecf601f26","Type":"ContainerStarted","Data":"3d85160fcd6b336644989c45a2f507fa4649ecd7f45d205f8b4cbc7f524183c4"} Nov 02 22:14:54.191414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:54.191229 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" event={"ID":"f2124f9b-a6c2-401d-99fd-8c0ecf601f26","Type":"ContainerStarted","Data":"afe14ab6092d795ad487f35730f2a870685343d27096e7203e5a93edd859581c"} Nov 02 22:14:54.304477 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:54.304419 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" podStartSLOduration=1.713903492 podStartE2EDuration="2.304403348s" totalImagesPullingTime="590.499856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:54.303396184 +0000 UTC m=+530.568389403" watchObservedRunningTime="2026-11-02 22:14:54.304403348 +0000 UTC m=+530.569396567" Nov 02 22:14:54.799061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:54.799027 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:14:55.881411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:55.881375 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:55.894133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:55.894109 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:56.211472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:14:56.211386 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8hsg9" Nov 02 22:14:56.495102 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:14:56.494969 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:15:00.167194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.167154 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg"] Nov 02 22:15:00.171403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.171377 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.187463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.187430 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e89062-4201-4e08-ad46-45e905e18fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-md8wg\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.187635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.187482 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/21e89062-4201-4e08-ad46-45e905e18fdf-kube-api-access-ccff6\") pod \"maas-api-key-cleanup-29894295-md8wg\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.225898 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.225861 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg"] Nov 02 22:15:00.270513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.270478 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:15:00.288532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.288499 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e89062-4201-4e08-ad46-45e905e18fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-md8wg\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.288738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.288608 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/21e89062-4201-4e08-ad46-45e905e18fdf-kube-api-access-ccff6\") pod \"maas-api-key-cleanup-29894295-md8wg\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.289145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.289124 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e89062-4201-4e08-ad46-45e905e18fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-md8wg\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.347156 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.347115 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/21e89062-4201-4e08-ad46-45e905e18fdf-kube-api-access-ccff6\") pod \"maas-api-key-cleanup-29894295-md8wg\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.484888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.484787 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:00.855266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:00.855078 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg"] Nov 02 22:15:00.858657 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:15:00.858629 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e89062_4201_4e08_ad46_45e905e18fdf.slice/crio-f1d164caab1592629011fbfd5e7de21a2c7a152072e207a67643a6c83ff8a792 WatchSource:0}: Error finding container f1d164caab1592629011fbfd5e7de21a2c7a152072e207a67643a6c83ff8a792: Status 404 returned error can't find the container with id f1d164caab1592629011fbfd5e7de21a2c7a152072e207a67643a6c83ff8a792 Nov 02 22:15:01.225163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:01.225078 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" event={"ID":"21e89062-4201-4e08-ad46-45e905e18fdf","Type":"ContainerStarted","Data":"f1d164caab1592629011fbfd5e7de21a2c7a152072e207a67643a6c83ff8a792"} Nov 02 22:15:02.798442 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:02.798405 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:15:02.811103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:02.811075 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:15:03.245530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:03.245441 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts" Nov 02 22:15:09.263273 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.263234 5109 generic.go:356] "Generic (PLEG): container finished" podID="21e89062-4201-4e08-ad46-45e905e18fdf" containerID="b72c30f54ac4b7d3ca7ab79ec5411049197f09a03a99506bfa67c953ac64a89f" exitCode=60 Nov 02 22:15:09.263689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.263275 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" event={"ID":"21e89062-4201-4e08-ad46-45e905e18fdf","Type":"ContainerDied","Data":"b72c30f54ac4b7d3ca7ab79ec5411049197f09a03a99506bfa67c953ac64a89f"} Nov 02 22:15:09.263689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.263601 5109 scope.go:122] "RemoveContainer" containerID="b72c30f54ac4b7d3ca7ab79ec5411049197f09a03a99506bfa67c953ac64a89f" Nov 02 22:15:09.629334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.629297 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk"] Nov 02 22:15:09.632910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.632894 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.676885 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.676847 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eafe82-f30e-429e-907e-0d60710314a8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.676894 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eafe82-f30e-429e-907e-0d60710314a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.676995 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.677055 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wm2\" (UniqueName: \"kubernetes.io/projected/c7eafe82-f30e-429e-907e-0d60710314a8-kube-api-access-p4wm2\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.677084 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.677109 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eafe82-f30e-429e-907e-0d60710314a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.677206 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.677303 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.677354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.677325 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.716304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.716244 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk"] Nov 02 22:15:09.778315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778280 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778313 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778344 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eafe82-f30e-429e-907e-0d60710314a8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778359 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eafe82-f30e-429e-907e-0d60710314a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778380 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778398 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wm2\" (UniqueName: \"kubernetes.io/projected/c7eafe82-f30e-429e-907e-0d60710314a8-kube-api-access-p4wm2\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778422 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778444 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eafe82-f30e-429e-907e-0d60710314a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778483 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778736 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778822 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.778974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.778843 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.779182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.779161 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eafe82-f30e-429e-907e-0d60710314a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.779385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.779364 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.781478 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.781451 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eafe82-f30e-429e-907e-0d60710314a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.781600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.781516 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eafe82-f30e-429e-907e-0d60710314a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.837059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.836986 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eafe82-f30e-429e-907e-0d60710314a8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.837059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.837030 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wm2\" (UniqueName: \"kubernetes.io/projected/c7eafe82-f30e-429e-907e-0d60710314a8-kube-api-access-p4wm2\") pod \"maas-default-gateway-openshift-default-76f4876667-9jkwk\" (UID: \"c7eafe82-f30e-429e-907e-0d60710314a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:09.945322 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:09.945287 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:10.270015 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.269983 5109 generic.go:356] "Generic (PLEG): container finished" podID="21e89062-4201-4e08-ad46-45e905e18fdf" containerID="f3da9a4c8e3bf34d9bf073250ac0463553b9e0ad3cd927921de27baa430b0bed" exitCode=60 Nov 02 22:15:10.270467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.270063 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" event={"ID":"21e89062-4201-4e08-ad46-45e905e18fdf","Type":"ContainerDied","Data":"f3da9a4c8e3bf34d9bf073250ac0463553b9e0ad3cd927921de27baa430b0bed"} Nov 02 22:15:10.270467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.270103 5109 scope.go:122] "RemoveContainer" containerID="b72c30f54ac4b7d3ca7ab79ec5411049197f09a03a99506bfa67c953ac64a89f" Nov 02 22:15:10.270467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.270387 5109 scope.go:122] "RemoveContainer" containerID="f3da9a4c8e3bf34d9bf073250ac0463553b9e0ad3cd927921de27baa430b0bed" Nov 02 22:15:10.270689 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:15:10.270662 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-md8wg_odh-ai-gateway-infra(21e89062-4201-4e08-ad46-45e905e18fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" Nov 02 22:15:10.365969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.365939 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk"] Nov 02 22:15:10.368507 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:15:10.368465 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eafe82_f30e_429e_907e_0d60710314a8.slice/crio-de56736e0a2ab7a253734bf12c7d53dfe821776a87e913e4cfcd8ea2babbbcff WatchSource:0}: Error finding container de56736e0a2ab7a253734bf12c7d53dfe821776a87e913e4cfcd8ea2babbbcff: Status 404 returned error can't find the container with id de56736e0a2ab7a253734bf12c7d53dfe821776a87e913e4cfcd8ea2babbbcff Nov 02 22:15:10.370945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.370910 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:15:10.371036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.370982 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:15:10.371036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:10.371012 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:15:11.277126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:11.277101 5109 scope.go:122] "RemoveContainer" containerID="f3da9a4c8e3bf34d9bf073250ac0463553b9e0ad3cd927921de27baa430b0bed" Nov 02 22:15:11.277618 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:15:11.277325 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-md8wg_odh-ai-gateway-infra(21e89062-4201-4e08-ad46-45e905e18fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" Nov 02 22:15:11.278524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:11.278501 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" event={"ID":"c7eafe82-f30e-429e-907e-0d60710314a8","Type":"ContainerStarted","Data":"2a35f3a914d92b53d10b17c839cf43c1bb2a82e2de3b7f4cba7f9620122744a0"} Nov 02 22:15:11.278644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:11.278543 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" event={"ID":"c7eafe82-f30e-429e-907e-0d60710314a8","Type":"ContainerStarted","Data":"de56736e0a2ab7a253734bf12c7d53dfe821776a87e913e4cfcd8ea2babbbcff"} Nov 02 22:15:11.521210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:11.521104 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podStartSLOduration=2.521089858 podStartE2EDuration="2.521089858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:11.519531845 +0000 UTC m=+547.784525063" watchObservedRunningTime="2026-11-02 22:15:11.521089858 +0000 UTC m=+547.786083076" Nov 02 22:15:11.945593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:11.945429 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:11.948744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:11.948711 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:15:11.949039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:11.948993 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 22:15:12.948964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:12.948921 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:15:12.949488 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:12.948981 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 22:15:13.949081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:13.949041 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:15:13.949622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:13.949116 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 22:15:14.948595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:14.947987 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:15:14.948595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:14.948071 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 22:15:15.948595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:15.948013 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:15:15.948595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:15.948084 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 22:15:16.946751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:16.946706 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:15:16.946920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:16.946777 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 22:15:17.946533 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:17.946489 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:15:17.946991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:17.946563 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 22:15:18.950068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:18.950037 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:18.950547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:18.950447 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:18.951128 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:18.951112 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" Nov 02 22:15:24.885208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:24.885166 5109 scope.go:122] "RemoveContainer" containerID="f3da9a4c8e3bf34d9bf073250ac0463553b9e0ad3cd927921de27baa430b0bed" Nov 02 22:15:25.346791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:25.346750 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" event={"ID":"21e89062-4201-4e08-ad46-45e905e18fdf","Type":"ContainerStarted","Data":"8ff92c545ec756952f7ee9a1d504378e8ebc1f648c318c0ffba613b61055a9b3"} Nov 02 22:15:25.459539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:25.459407 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" podStartSLOduration=17.853142295 podStartE2EDuration="25.459392229s" totalImagesPullingTime="7.606249934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:25.459243503 +0000 UTC m=+561.724236721" watchObservedRunningTime="2026-11-02 22:15:25.459392229 +0000 UTC m=+561.724385447" Nov 02 22:15:26.353515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:26.353480 5109 generic.go:356] "Generic (PLEG): container finished" podID="21e89062-4201-4e08-ad46-45e905e18fdf" containerID="8ff92c545ec756952f7ee9a1d504378e8ebc1f648c318c0ffba613b61055a9b3" exitCode=60 Nov 02 22:15:26.353976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:26.353551 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" event={"ID":"21e89062-4201-4e08-ad46-45e905e18fdf","Type":"ContainerDied","Data":"8ff92c545ec756952f7ee9a1d504378e8ebc1f648c318c0ffba613b61055a9b3"} Nov 02 22:15:26.353976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:26.353634 5109 scope.go:122] "RemoveContainer" containerID="f3da9a4c8e3bf34d9bf073250ac0463553b9e0ad3cd927921de27baa430b0bed" Nov 02 22:15:26.353976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:26.353917 5109 scope.go:122] "RemoveContainer" containerID="8ff92c545ec756952f7ee9a1d504378e8ebc1f648c318c0ffba613b61055a9b3" Nov 02 22:15:26.354179 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:15:26.354162 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-md8wg_odh-ai-gateway-infra(21e89062-4201-4e08-ad46-45e905e18fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" Nov 02 22:15:26.473436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:26.473400 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg"] Nov 02 22:15:26.491929 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:15:26.491892 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:15:27.509066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:27.509031 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:27.672713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:27.672627 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e89062-4201-4e08-ad46-45e905e18fdf-kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/21e89062-4201-4e08-ad46-45e905e18fdf-kube-api-access-ccff6\") pod \"21e89062-4201-4e08-ad46-45e905e18fdf\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " Nov 02 22:15:27.672713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:27.672681 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e89062-4201-4e08-ad46-45e905e18fdf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e89062-4201-4e08-ad46-45e905e18fdf-maas-api-service-ca\") pod \"21e89062-4201-4e08-ad46-45e905e18fdf\" (UID: \"21e89062-4201-4e08-ad46-45e905e18fdf\") " Nov 02 22:15:27.673124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:27.673096 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e89062-4201-4e08-ad46-45e905e18fdf-maas-api-service-ca" pod "21e89062-4201-4e08-ad46-45e905e18fdf" (UID: "21e89062-4201-4e08-ad46-45e905e18fdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:27.675071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:27.675049 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e89062-4201-4e08-ad46-45e905e18fdf-kube-api-access-ccff6" pod "21e89062-4201-4e08-ad46-45e905e18fdf" (UID: "21e89062-4201-4e08-ad46-45e905e18fdf"). InnerVolumeSpecName "kube-api-access-ccff6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:27.773869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:27.773833 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccff6\" (UniqueName: \"kubernetes.io/projected/21e89062-4201-4e08-ad46-45e905e18fdf-kube-api-access-ccff6\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:27.773869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:27.773861 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e89062-4201-4e08-ad46-45e905e18fdf-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:28.365422 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:28.365385 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" event={"ID":"21e89062-4201-4e08-ad46-45e905e18fdf","Type":"ContainerDied","Data":"f1d164caab1592629011fbfd5e7de21a2c7a152072e207a67643a6c83ff8a792"} Nov 02 22:15:28.365671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:28.365430 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg" Nov 02 22:15:28.365671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:28.365448 5109 scope.go:122] "RemoveContainer" containerID="8ff92c545ec756952f7ee9a1d504378e8ebc1f648c318c0ffba613b61055a9b3" Nov 02 22:15:28.524242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:28.524197 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg"] Nov 02 22:15:28.528761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:28.528732 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-md8wg"] Nov 02 22:15:29.890018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:29.889982 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" path="/var/lib/kubelet/pods/21e89062-4201-4e08-ad46-45e905e18fdf/volumes" Nov 02 22:15:31.161494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.161456 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw"] Nov 02 22:15:31.161905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.161846 5109 cpu_manager.go:414] "containerMap: removing container" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.161905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.161857 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.161905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.161877 5109 cpu_manager.go:414] "containerMap: removing container" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.161905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.161882 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.162033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.161968 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.162033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.161979 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.165154 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.165132 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.205283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.205246 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqsq\" (UniqueName: \"kubernetes.io/projected/72f8a56d-ed2f-4fc9-a5f5-ab414e384512-kube-api-access-9xqsq\") pod \"payload-pre-processing-7b98bb678c-9vjjw\" (UID: \"72f8a56d-ed2f-4fc9-a5f5-ab414e384512\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.205283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.205292 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f8a56d-ed2f-4fc9-a5f5-ab414e384512-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9vjjw\" (UID: \"72f8a56d-ed2f-4fc9-a5f5-ab414e384512\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.215256 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.215230 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 22:15:31.261997 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.261960 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kbwhc\"" Nov 02 22:15:31.263385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.263355 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw"] Nov 02 22:15:31.280932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.280898 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vng6p"] Nov 02 22:15:31.281344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.281319 5109 cpu_manager.go:414] "containerMap: removing container" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.281344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.281339 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.281633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.281418 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e89062-4201-4e08-ad46-45e905e18fdf" containerName="cleanup" Nov 02 22:15:31.284673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.284653 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.306601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.306555 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rj2\" (UniqueName: \"kubernetes.io/projected/5ce52112-b66d-4857-921c-ffcde641d781-kube-api-access-v5rj2\") pod \"payload-processing-55fbbbf9dc-vng6p\" (UID: \"5ce52112-b66d-4857-921c-ffcde641d781\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.306767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.306687 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce52112-b66d-4857-921c-ffcde641d781-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vng6p\" (UID: \"5ce52112-b66d-4857-921c-ffcde641d781\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.306820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.306792 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqsq\" (UniqueName: \"kubernetes.io/projected/72f8a56d-ed2f-4fc9-a5f5-ab414e384512-kube-api-access-9xqsq\") pod \"payload-pre-processing-7b98bb678c-9vjjw\" (UID: \"72f8a56d-ed2f-4fc9-a5f5-ab414e384512\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.306873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.306829 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f8a56d-ed2f-4fc9-a5f5-ab414e384512-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9vjjw\" (UID: \"72f8a56d-ed2f-4fc9-a5f5-ab414e384512\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.307673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.307650 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f8a56d-ed2f-4fc9-a5f5-ab414e384512-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9vjjw\" (UID: \"72f8a56d-ed2f-4fc9-a5f5-ab414e384512\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.340815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.340766 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vng6p"] Nov 02 22:15:31.366515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.366479 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqsq\" (UniqueName: \"kubernetes.io/projected/72f8a56d-ed2f-4fc9-a5f5-ab414e384512-kube-api-access-9xqsq\") pod \"payload-pre-processing-7b98bb678c-9vjjw\" (UID: \"72f8a56d-ed2f-4fc9-a5f5-ab414e384512\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.407775 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.407732 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rj2\" (UniqueName: \"kubernetes.io/projected/5ce52112-b66d-4857-921c-ffcde641d781-kube-api-access-v5rj2\") pod \"payload-processing-55fbbbf9dc-vng6p\" (UID: \"5ce52112-b66d-4857-921c-ffcde641d781\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.407974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.407793 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce52112-b66d-4857-921c-ffcde641d781-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vng6p\" (UID: \"5ce52112-b66d-4857-921c-ffcde641d781\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.408495 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.408474 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce52112-b66d-4857-921c-ffcde641d781-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vng6p\" (UID: \"5ce52112-b66d-4857-921c-ffcde641d781\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.466136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.466050 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rj2\" (UniqueName: \"kubernetes.io/projected/5ce52112-b66d-4857-921c-ffcde641d781-kube-api-access-v5rj2\") pod \"payload-processing-55fbbbf9dc-vng6p\" (UID: \"5ce52112-b66d-4857-921c-ffcde641d781\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.478937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.478906 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:31.602010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.601965 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:31.850165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.850132 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw"] Nov 02 22:15:31.853466 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:15:31.853432 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f8a56d_ed2f_4fc9_a5f5_ab414e384512.slice/crio-923b3681d045f0f8c2da5c90ebb0d9a25c6f45fc6b00e29d234021ecdbb0cc38 WatchSource:0}: Error finding container 923b3681d045f0f8c2da5c90ebb0d9a25c6f45fc6b00e29d234021ecdbb0cc38: Status 404 returned error can't find the container with id 923b3681d045f0f8c2da5c90ebb0d9a25c6f45fc6b00e29d234021ecdbb0cc38 Nov 02 22:15:31.972312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:31.972282 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vng6p"] Nov 02 22:15:31.974555 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:15:31.974522 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce52112_b66d_4857_921c_ffcde641d781.slice/crio-e368c88059de89e011dc15f2cc97076a3ba251130d430b964f070bcb0c54ea00 WatchSource:0}: Error finding container e368c88059de89e011dc15f2cc97076a3ba251130d430b964f070bcb0c54ea00: Status 404 returned error can't find the container with id e368c88059de89e011dc15f2cc97076a3ba251130d430b964f070bcb0c54ea00 Nov 02 22:15:32.389210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:32.389175 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" event={"ID":"72f8a56d-ed2f-4fc9-a5f5-ab414e384512","Type":"ContainerStarted","Data":"923b3681d045f0f8c2da5c90ebb0d9a25c6f45fc6b00e29d234021ecdbb0cc38"} Nov 02 22:15:32.390880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:32.390841 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" event={"ID":"5ce52112-b66d-4857-921c-ffcde641d781","Type":"ContainerStarted","Data":"e368c88059de89e011dc15f2cc97076a3ba251130d430b964f070bcb0c54ea00"} Nov 02 22:15:35.413492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:35.413447 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" event={"ID":"72f8a56d-ed2f-4fc9-a5f5-ab414e384512","Type":"ContainerStarted","Data":"3f10e29f619c653d345f94eeddb9e79ccc119ea7838916fad4d807f2fdef962e"} Nov 02 22:15:35.413989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:35.413558 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:35.414991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:35.414963 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" event={"ID":"5ce52112-b66d-4857-921c-ffcde641d781","Type":"ContainerStarted","Data":"c315c70fe765291e3021f3ba868f1b780945fa06f66ce7aa59dad6213c4bce22"} Nov 02 22:15:35.415112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:35.415098 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:35.523415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:35.523352 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" podStartSLOduration=1.358221468 podStartE2EDuration="4.523338747s" totalImagesPullingTime="3.165117279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:35.522461258 +0000 UTC m=+571.787454477" watchObservedRunningTime="2026-11-02 22:15:35.523338747 +0000 UTC m=+571.788331966" Nov 02 22:15:35.633493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:35.633376 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" podStartSLOduration=1.552683838 podStartE2EDuration="4.633357479s" totalImagesPullingTime="3.080673641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:35.631998247 +0000 UTC m=+571.896991478" watchObservedRunningTime="2026-11-02 22:15:35.633357479 +0000 UTC m=+571.898350699" Nov 02 22:15:46.423010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:46.422978 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vng6p" Nov 02 22:15:46.423407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:46.423043 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vjjw" Nov 02 22:15:54.982106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:54.982072 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd"] Nov 02 22:15:54.985905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:54.985880 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.083601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.083543 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd"] Nov 02 22:15:55.130020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.129983 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130030 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130059 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130216 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130253 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13a2f99-892e-46ca-8ba2-9acd657898bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130273 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2qc\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-kube-api-access-zf2qc\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130386 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130421 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.130508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.130440 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231232 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231274 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13a2f99-892e-46ca-8ba2-9acd657898bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231292 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2qc\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-kube-api-access-zf2qc\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231328 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231358 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231376 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231416 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231448 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231479 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231753 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231854 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.231881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.231867 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.232103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.232017 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.232181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.232122 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13a2f99-892e-46ca-8ba2-9acd657898bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.233823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.233801 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.234033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.234014 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.289597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.289531 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.289754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.289652 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2qc\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-kube-api-access-zf2qc\") pod \"maas-default-gateway-openshift-default-76f4876667-zzqfd\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.298246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.298221 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:55.717296 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.717269 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd"] Nov 02 22:15:55.719454 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:15:55.719422 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13a2f99_892e_46ca_8ba2_9acd657898bf.slice/crio-a28e5edee307e3223971cf48f1cb9d29ff718f29cc0c12ea47e32bd672247776 WatchSource:0}: Error finding container a28e5edee307e3223971cf48f1cb9d29ff718f29cc0c12ea47e32bd672247776: Status 404 returned error can't find the container with id a28e5edee307e3223971cf48f1cb9d29ff718f29cc0c12ea47e32bd672247776 Nov 02 22:15:55.721769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.721735 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:15:55.721892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.721813 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:15:55.721892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:55.721859 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:15:56.490604 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:15:56.490555 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:15:56.521065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:56.521021 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" event={"ID":"c13a2f99-892e-46ca-8ba2-9acd657898bf","Type":"ContainerStarted","Data":"019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b"} Nov 02 22:15:56.521237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:56.521096 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" event={"ID":"c13a2f99-892e-46ca-8ba2-9acd657898bf","Type":"ContainerStarted","Data":"a28e5edee307e3223971cf48f1cb9d29ff718f29cc0c12ea47e32bd672247776"} Nov 02 22:15:56.739598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:56.737308 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podStartSLOduration=2.737284231 podStartE2EDuration="2.737284231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:56.680238874 +0000 UTC m=+592.945232090" watchObservedRunningTime="2026-11-02 22:15:56.737284231 +0000 UTC m=+593.002277451" Nov 02 22:15:57.298863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:57.298821 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:15:57.300350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:57.300321 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-zzqfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:15:57.300484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:57.300371 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:15:58.299725 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:58.299688 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-zzqfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:15:58.300199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:58.299749 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:15:59.299533 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:59.299497 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-zzqfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:15:59.299720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:15:59.299550 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:16:00.299149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:00.299109 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-zzqfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:16:00.299554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:00.299167 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:16:01.299463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:01.299422 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-zzqfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:16:01.299990 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:01.299483 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:16:02.299179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:02.299140 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-zzqfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:16:02.299355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:02.299227 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:16:04.080628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:04.080594 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:16:04.081354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:04.081333 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:16:04.082173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:04.082153 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:16:26.489943 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:16:26.489904 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:16:35.766490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.766448 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl"] Nov 02 22:16:35.770446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.770422 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.824214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.824185 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-711bd4-openshift-default-dockercfg-h26gm\"" Nov 02 22:16:35.825084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825057 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24mkp\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-kube-api-access-24mkp\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825103 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825125 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825200 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825266 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7539982-c036-4526-9d1f-bc9324bed23f-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825341 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825396 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825419 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7539982-c036-4526-9d1f-bc9324bed23f-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.825709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.825543 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.867998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.867963 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl"] Nov 02 22:16:35.926247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926210 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7539982-c036-4526-9d1f-bc9324bed23f-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926278 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926340 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926367 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7539982-c036-4526-9d1f-bc9324bed23f-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926400 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926452 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24mkp\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-kube-api-access-24mkp\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926498 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926521 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.926787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926544 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.927006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.926978 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.927092 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.927013 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.927310 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.927280 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.927432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.927385 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.927698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.927443 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7539982-c036-4526-9d1f-bc9324bed23f-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.929550 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.929529 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7539982-c036-4526-9d1f-bc9324bed23f-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.930013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.929992 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.984873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.984839 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:35.985052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:35.985028 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24mkp\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-kube-api-access-24mkp\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:36.083505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.083416 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:36.437134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.437054 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv"] Nov 02 22:16:36.441832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.441808 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.493292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.493263 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fe810f-openshift-default-dockercfg-lk5vp\"" Nov 02 22:16:36.500418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.500386 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv"] Nov 02 22:16:36.532394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532363 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532415 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ec1893-7330-4af0-91d1-448ba3809d8c-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532436 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpnh\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-kube-api-access-mfpnh\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532546 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532617 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532667 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532714 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532742 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.532903 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.532775 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.568506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.568476 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl"] Nov 02 22:16:36.571622 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:16:36.571595 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7539982_c036_4526_9d1f_bc9324bed23f.slice/crio-1baa195c76b5bbab0e01e769dab2378607871b7008c51942e15d2d1fb446a578 WatchSource:0}: Error finding container 1baa195c76b5bbab0e01e769dab2378607871b7008c51942e15d2d1fb446a578: Status 404 returned error can't find the container with id 1baa195c76b5bbab0e01e769dab2378607871b7008c51942e15d2d1fb446a578 Nov 02 22:16:36.574085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.574052 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:36.574153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.574144 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:36.574196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.574174 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:36.633856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.633830 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.633992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.633865 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.633992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.633918 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.633992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.633949 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.633992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.633986 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634036 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634081 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634153 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ec1893-7330-4af0-91d1-448ba3809d8c-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634186 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpnh\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-kube-api-access-mfpnh\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634266 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634348 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634488 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634400 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634732 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.634921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.634902 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ec1893-7330-4af0-91d1-448ba3809d8c-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.636258 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.636240 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.636465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.636447 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.692609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.692486 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.692609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.692548 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpnh\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-kube-api-access-mfpnh\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.733308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.733262 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" event={"ID":"d7539982-c036-4526-9d1f-bc9324bed23f","Type":"ContainerStarted","Data":"bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d"} Nov 02 22:16:36.733497 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.733322 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" event={"ID":"d7539982-c036-4526-9d1f-bc9324bed23f","Type":"ContainerStarted","Data":"1baa195c76b5bbab0e01e769dab2378607871b7008c51942e15d2d1fb446a578"} Nov 02 22:16:36.755225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.755194 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:36.850726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:36.850661 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podStartSLOduration=1.850638944 podStartE2EDuration="1.850638944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:36.848973714 +0000 UTC m=+633.113966933" watchObservedRunningTime="2026-11-02 22:16:36.850638944 +0000 UTC m=+633.115632166" Nov 02 22:16:37.083734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.083694 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:37.089556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.089524 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:37.185770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.185745 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv"] Nov 02 22:16:37.187839 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:16:37.187805 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ec1893_7330_4af0_91d1_448ba3809d8c.slice/crio-fa3b98fc1e68550b3a0fd1df3a21585fc7d9230880545df1d27dcc2bc364415a WatchSource:0}: Error finding container fa3b98fc1e68550b3a0fd1df3a21585fc7d9230880545df1d27dcc2bc364415a: Status 404 returned error can't find the container with id fa3b98fc1e68550b3a0fd1df3a21585fc7d9230880545df1d27dcc2bc364415a Nov 02 22:16:37.190373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.190340 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:37.190468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.190411 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:37.190468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.190441 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:37.411941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.411842 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q"] Nov 02 22:16:37.416199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.416176 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.476867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.476550 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q"] Nov 02 22:16:37.548591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.548524 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ca85db-openshift-default-dockercfg-bzw5r\"" Nov 02 22:16:37.553896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.553869 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554003 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554044 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554145 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d2d9c2-5758-4033-a9b8-26e322a0af11-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554209 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554247 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554301 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554390 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554334 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.554390 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.554366 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jftbc\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-kube-api-access-jftbc\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.655839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.655801 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d2d9c2-5758-4033-a9b8-26e322a0af11-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.655839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.655838 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.655861 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.655876 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.655892 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.655912 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftbc\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-kube-api-access-jftbc\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.655995 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.656081 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.656107 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.656320 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.656433 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.656451 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.656456 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.656904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.656883 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d2d9c2-5758-4033-a9b8-26e322a0af11-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.658542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.658524 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.658772 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.658753 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.717217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.717148 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.717347 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.717302 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jftbc\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-kube-api-access-jftbc\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.729409 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.729382 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:37.739593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.739529 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" event={"ID":"23ec1893-7330-4af0-91d1-448ba3809d8c","Type":"ContainerStarted","Data":"89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f"} Nov 02 22:16:37.739726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.739614 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" event={"ID":"23ec1893-7330-4af0-91d1-448ba3809d8c","Type":"ContainerStarted","Data":"fa3b98fc1e68550b3a0fd1df3a21585fc7d9230880545df1d27dcc2bc364415a"} Nov 02 22:16:37.740051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.740033 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:37.741086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.741057 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:16:37.756204 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.756173 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:37.857412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:37.857359 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podStartSLOduration=1.85734497 podStartE2EDuration="1.85734497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:37.855595411 +0000 UTC m=+634.120588628" watchObservedRunningTime="2026-11-02 22:16:37.85734497 +0000 UTC m=+634.122338188" Nov 02 22:16:38.167841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.167801 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q"] Nov 02 22:16:38.183639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.183380 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:38.183639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.183473 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:38.183639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.183520 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:16:38.746413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.746376 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" event={"ID":"55d2d9c2-5758-4033-a9b8-26e322a0af11","Type":"ContainerStarted","Data":"877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a"} Nov 02 22:16:38.746609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.746438 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" event={"ID":"55d2d9c2-5758-4033-a9b8-26e322a0af11","Type":"ContainerStarted","Data":"7adcf0ff63e51af02503d3667b30619fa9fa5e473c02ca2c4058d98edbd0e02a"} Nov 02 22:16:38.762492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.762462 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:38.878693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:38.878645 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podStartSLOduration=1.8786315999999998 podStartE2EDuration="1.8786316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:38.878143956 +0000 UTC m=+635.143137199" watchObservedRunningTime="2026-11-02 22:16:38.8786316 +0000 UTC m=+635.143624819" Nov 02 22:16:39.729841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:39.729802 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:39.735617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:39.735562 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:39.751651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:39.751622 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:39.751893 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:39.751869 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:39.752314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:39.752293 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:16:39.752397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:39.752341 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:16:52.871422 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:52.871328 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g"] Nov 02 22:16:52.875096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:52.875074 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:52.938695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:52.938658 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9km\" (UniqueName: \"kubernetes.io/projected/4da2ea58-8933-496e-a7d2-7711d131c55d-kube-api-access-ts9km\") pod \"payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:52.938879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:52.938729 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da2ea58-8933-496e-a7d2-7711d131c55d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:52.971845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:52.971809 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-49a557-dockercfg-v9l6w\"" Nov 02 22:16:52.971845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:52.971809 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-49a557\"" Nov 02 22:16:52.973735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:52.973706 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g"] Nov 02 22:16:53.033292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.033252 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c"] Nov 02 22:16:53.040198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.040163 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da2ea58-8933-496e-a7d2-7711d131c55d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:53.040384 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.040334 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9km\" (UniqueName: \"kubernetes.io/projected/4da2ea58-8933-496e-a7d2-7711d131c55d-kube-api-access-ts9km\") pod \"payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:53.041469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.041446 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da2ea58-8933-496e-a7d2-7711d131c55d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:53.044813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.044016 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.089996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.089961 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c"] Nov 02 22:16:53.099016 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.098986 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9km\" (UniqueName: \"kubernetes.io/projected/4da2ea58-8933-496e-a7d2-7711d131c55d-kube-api-access-ts9km\") pod \"payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:53.141223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.141117 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500893a0-a2f3-4840-b9ee-59e5226d9335-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.141223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.141170 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsfx\" (UniqueName: \"kubernetes.io/projected/500893a0-a2f3-4840-b9ee-59e5226d9335-kube-api-access-wmsfx\") pod \"payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.188370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.188329 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:53.242920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.241884 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsfx\" (UniqueName: \"kubernetes.io/projected/500893a0-a2f3-4840-b9ee-59e5226d9335-kube-api-access-wmsfx\") pod \"payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.242920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.242055 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500893a0-a2f3-4840-b9ee-59e5226d9335-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.243321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.243295 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500893a0-a2f3-4840-b9ee-59e5226d9335-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.300598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.300546 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsfx\" (UniqueName: \"kubernetes.io/projected/500893a0-a2f3-4840-b9ee-59e5226d9335-kube-api-access-wmsfx\") pod \"payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.358813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.358782 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.557768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.557742 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g"] Nov 02 22:16:53.559960 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:16:53.559907 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da2ea58_8933_496e_a7d2_7711d131c55d.slice/crio-dbfa6799032d45d90f406171347d721dd06b95db781dfe419924609cb163df68 WatchSource:0}: Error finding container dbfa6799032d45d90f406171347d721dd06b95db781dfe419924609cb163df68: Status 404 returned error can't find the container with id dbfa6799032d45d90f406171347d721dd06b95db781dfe419924609cb163df68 Nov 02 22:16:53.730129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.730095 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c"] Nov 02 22:16:53.731383 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:16:53.731346 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500893a0_a2f3_4840_b9ee_59e5226d9335.slice/crio-7ad27238fd6e25a66aa59f45bb9b5aebdc052122e4cfd70684139befe15a16b1 WatchSource:0}: Error finding container 7ad27238fd6e25a66aa59f45bb9b5aebdc052122e4cfd70684139befe15a16b1: Status 404 returned error can't find the container with id 7ad27238fd6e25a66aa59f45bb9b5aebdc052122e4cfd70684139befe15a16b1 Nov 02 22:16:53.830923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.830878 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" event={"ID":"4da2ea58-8933-496e-a7d2-7711d131c55d","Type":"ContainerStarted","Data":"37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5"} Nov 02 22:16:53.831121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.831041 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:16:53.831121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.831087 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" event={"ID":"4da2ea58-8933-496e-a7d2-7711d131c55d","Type":"ContainerStarted","Data":"dbfa6799032d45d90f406171347d721dd06b95db781dfe419924609cb163df68"} Nov 02 22:16:53.832847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.832819 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" event={"ID":"500893a0-a2f3-4840-b9ee-59e5226d9335","Type":"ContainerStarted","Data":"021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151"} Nov 02 22:16:53.832966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.832859 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" event={"ID":"500893a0-a2f3-4840-b9ee-59e5226d9335","Type":"ContainerStarted","Data":"7ad27238fd6e25a66aa59f45bb9b5aebdc052122e4cfd70684139befe15a16b1"} Nov 02 22:16:53.832966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.832915 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:16:53.944259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:53.944191 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" podStartSLOduration=1.944172671 podStartE2EDuration="1.944172671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:53.942203825 +0000 UTC m=+650.207197044" watchObservedRunningTime="2026-11-02 22:16:53.944172671 +0000 UTC m=+650.209165890" Nov 02 22:16:54.054457 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:54.054398 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" podStartSLOduration=2.054381356 podStartE2EDuration="2.054381356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:54.052143853 +0000 UTC m=+650.317137094" watchObservedRunningTime="2026-11-02 22:16:54.054381356 +0000 UTC m=+650.319374575" Nov 02 22:16:56.490098 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:16:56.490063 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:16:58.281382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.281349 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr"] Nov 02 22:16:58.291151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.291116 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.341359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.341316 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr"] Nov 02 22:16:58.388072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.385917 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-fe810f-dockercfg-fqg7z\"" Nov 02 22:16:58.388072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.386039 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-fe810f\"" Nov 02 22:16:58.411293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.410342 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkw5n\" (UniqueName: \"kubernetes.io/projected/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-kube-api-access-qkw5n\") pod \"payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.411293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.410420 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.415176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.415132 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8"] Nov 02 22:16:58.419430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.419401 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.477281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.477246 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8"] Nov 02 22:16:58.511672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.511634 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/353686ce-7828-478a-b4ad-9f27c3fdc577-kube-api-access-8jhbm\") pod \"payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.511845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.511685 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkw5n\" (UniqueName: \"kubernetes.io/projected/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-kube-api-access-qkw5n\") pod \"payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.511845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.511756 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.511845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.511800 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353686ce-7828-478a-b4ad-9f27c3fdc577-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.512469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.512450 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.570107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.570027 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkw5n\" (UniqueName: \"kubernetes.io/projected/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-kube-api-access-qkw5n\") pod \"payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.606086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.606045 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:58.612660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.612632 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/353686ce-7828-478a-b4ad-9f27c3fdc577-kube-api-access-8jhbm\") pod \"payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.612798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.612709 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353686ce-7828-478a-b4ad-9f27c3fdc577-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.613372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.613354 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353686ce-7828-478a-b4ad-9f27c3fdc577-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.671453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.671379 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/353686ce-7828-478a-b4ad-9f27c3fdc577-kube-api-access-8jhbm\") pod \"payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.733252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.733208 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:58.974157 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:58.974129 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr"] Nov 02 22:16:58.976629 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:16:58.976599 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4ea3c9_b253_455c_b55d_c1f9c2b0b385.slice/crio-0a6a5e3cb22f7baf7c31c9a1e85e93c157b53abf9b9c0d4af7654bacb3269b9d WatchSource:0}: Error finding container 0a6a5e3cb22f7baf7c31c9a1e85e93c157b53abf9b9c0d4af7654bacb3269b9d: Status 404 returned error can't find the container with id 0a6a5e3cb22f7baf7c31c9a1e85e93c157b53abf9b9c0d4af7654bacb3269b9d Nov 02 22:16:59.105295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.105257 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8"] Nov 02 22:16:59.864755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.864720 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" event={"ID":"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385","Type":"ContainerStarted","Data":"613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029"} Nov 02 22:16:59.864755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.864774 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" event={"ID":"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385","Type":"ContainerStarted","Data":"0a6a5e3cb22f7baf7c31c9a1e85e93c157b53abf9b9c0d4af7654bacb3269b9d"} Nov 02 22:16:59.865388 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.864948 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:16:59.866387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.866361 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" event={"ID":"353686ce-7828-478a-b4ad-9f27c3fdc577","Type":"ContainerStarted","Data":"b3b836ee829d9e345cdb67d4fe1918f37764e50313f4039608bf6ae147fb7a38"} Nov 02 22:16:59.866387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.866398 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" event={"ID":"353686ce-7828-478a-b4ad-9f27c3fdc577","Type":"ContainerStarted","Data":"a0fd26085e095cfdd56030b177d2d41b0566c024986c2d7fdaf465446889e13c"} Nov 02 22:16:59.866950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.866499 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:16:59.976703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:16:59.976646 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" podStartSLOduration=1.976628748 podStartE2EDuration="1.976628748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:59.974058607 +0000 UTC m=+656.239051826" watchObservedRunningTime="2026-11-02 22:16:59.976628748 +0000 UTC m=+656.241621969" Nov 02 22:17:00.085882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:00.084842 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" podStartSLOduration=2.08482293 podStartE2EDuration="2.08482293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:00.084201354 +0000 UTC m=+656.349194578" watchObservedRunningTime="2026-11-02 22:17:00.08482293 +0000 UTC m=+656.349816152" Nov 02 22:17:03.228494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.228455 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn"] Nov 02 22:17:03.259592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.251511 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.287317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.287277 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn"] Nov 02 22:17:03.300987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.300954 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3224ea\"" Nov 02 22:17:03.383090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.383043 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.383278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.383257 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8tl\" (UniqueName: \"kubernetes.io/projected/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-kube-api-access-qn8tl\") pod \"payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.484414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.484292 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8tl\" (UniqueName: \"kubernetes.io/projected/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-kube-api-access-qn8tl\") pod \"payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.484414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.484386 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.485123 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.485104 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.542222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.542184 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8tl\" (UniqueName: \"kubernetes.io/projected/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-kube-api-access-qn8tl\") pod \"payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.576244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.576194 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:03.946308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:03.946283 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn"] Nov 02 22:17:03.948506 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:17:03.948471 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cad0f9_cd2a_42e7_b4fb_5260667ec171.slice/crio-52078d3e560c3356e9e2340f8dd54060e9e37ef04d6bbb4d49639879e9a3f8f7 WatchSource:0}: Error finding container 52078d3e560c3356e9e2340f8dd54060e9e37ef04d6bbb4d49639879e9a3f8f7: Status 404 returned error can't find the container with id 52078d3e560c3356e9e2340f8dd54060e9e37ef04d6bbb4d49639879e9a3f8f7 Nov 02 22:17:04.849685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:04.849653 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:17:04.850095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:04.849726 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:17:04.899967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:04.899928 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" event={"ID":"f9cad0f9-cd2a-42e7-b4fb-5260667ec171","Type":"ContainerStarted","Data":"8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71"} Nov 02 22:17:04.900164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:04.899995 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" event={"ID":"f9cad0f9-cd2a-42e7-b4fb-5260667ec171","Type":"ContainerStarted","Data":"52078d3e560c3356e9e2340f8dd54060e9e37ef04d6bbb4d49639879e9a3f8f7"} Nov 02 22:17:04.900164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:04.900142 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:05.182405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:05.182265 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" podStartSLOduration=2.182245205 podStartE2EDuration="2.182245205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:05.179024075 +0000 UTC m=+661.444017296" watchObservedRunningTime="2026-11-02 22:17:05.182245205 +0000 UTC m=+661.447238423" Nov 02 22:17:10.875356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:10.875320 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:17:10.875862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:10.875408 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:17:13.120720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.120673 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww"] Nov 02 22:17:13.126290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.126259 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.179888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.179802 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww"] Nov 02 22:17:13.235628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.224289 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.236091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.236058 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clhzj\" (UniqueName: \"kubernetes.io/projected/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-kube-api-access-clhzj\") pod \"payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.236666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.236642 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-711bd4-dockercfg-ntbvz\"" Nov 02 22:17:13.239434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.239357 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-711bd4\"" Nov 02 22:17:13.276940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.276893 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx"] Nov 02 22:17:13.281477 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.281448 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.338041 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.337980 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clhzj\" (UniqueName: \"kubernetes.io/projected/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-kube-api-access-clhzj\") pod \"payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.338302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.338110 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.339018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.338995 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.355553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.355518 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx"] Nov 02 22:17:13.429826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.429744 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clhzj\" (UniqueName: \"kubernetes.io/projected/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-kube-api-access-clhzj\") pod \"payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.439638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.439609 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkvs\" (UniqueName: \"kubernetes.io/projected/e3b3e669-f53f-4070-bbc7-408628639dca-kube-api-access-gvkvs\") pod \"payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.439836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.439813 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b3e669-f53f-4070-bbc7-408628639dca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.443411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.443395 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.541475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.541428 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b3e669-f53f-4070-bbc7-408628639dca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.541695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.541526 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkvs\" (UniqueName: \"kubernetes.io/projected/e3b3e669-f53f-4070-bbc7-408628639dca-kube-api-access-gvkvs\") pod \"payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.542210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.542187 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b3e669-f53f-4070-bbc7-408628639dca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.599660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.599626 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkvs\" (UniqueName: \"kubernetes.io/projected/e3b3e669-f53f-4070-bbc7-408628639dca-kube-api-access-gvkvs\") pod \"payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.603472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.603440 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:13.829452 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.829426 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww"] Nov 02 22:17:13.831149 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:17:13.831119 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6cc8e5_2298_4660_8b87_d9c23fcdccac.slice/crio-4c82371bc6ba251e3facdeef8a8742ed37235be408813d7f3d56776e43774a77 WatchSource:0}: Error finding container 4c82371bc6ba251e3facdeef8a8742ed37235be408813d7f3d56776e43774a77: Status 404 returned error can't find the container with id 4c82371bc6ba251e3facdeef8a8742ed37235be408813d7f3d56776e43774a77 Nov 02 22:17:13.977271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.977123 5109 generic.go:356] "Generic (PLEG): container finished" podID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerID="89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f" exitCode=0 Nov 02 22:17:13.977271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.977208 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" event={"ID":"23ec1893-7330-4af0-91d1-448ba3809d8c","Type":"ContainerDied","Data":"89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f"} Nov 02 22:17:13.977840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.977727 5109 scope.go:122] "RemoveContainer" containerID="89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f" Nov 02 22:17:13.978051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.978026 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx"] Nov 02 22:17:13.979115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.979096 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" event={"ID":"6a6cc8e5-2298-4660-8b87-d9c23fcdccac","Type":"ContainerStarted","Data":"ee376daaa0bbdeb912e64220da5f734e7617ccc730d5173f6864c8166ae3c23b"} Nov 02 22:17:13.979194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.979135 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" event={"ID":"6a6cc8e5-2298-4660-8b87-d9c23fcdccac","Type":"ContainerStarted","Data":"4c82371bc6ba251e3facdeef8a8742ed37235be408813d7f3d56776e43774a77"} Nov 02 22:17:13.979242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.979206 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:13.980105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.980077 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:13.980174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.980140 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:13.980222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:13.980170 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:13.980623 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:17:13.980600 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b3e669_f53f_4070_bbc7_408628639dca.slice/crio-7f58c7bfce8aae581580c68c8a22793dc4e9a7f59be530b895c045eaca1f78c0 WatchSource:0}: Error finding container 7f58c7bfce8aae581580c68c8a22793dc4e9a7f59be530b895c045eaca1f78c0: Status 404 returned error can't find the container with id 7f58c7bfce8aae581580c68c8a22793dc4e9a7f59be530b895c045eaca1f78c0 Nov 02 22:17:14.205069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:14.204948 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" podStartSLOduration=1.204929748 podStartE2EDuration="1.204929748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:14.202077013 +0000 UTC m=+670.467070243" watchObservedRunningTime="2026-11-02 22:17:14.204929748 +0000 UTC m=+670.469922965" Nov 02 22:17:14.985673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:14.985633 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" event={"ID":"e3b3e669-f53f-4070-bbc7-408628639dca","Type":"ContainerStarted","Data":"cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10"} Nov 02 22:17:14.985857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:14.985836 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:14.985948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:14.985872 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" event={"ID":"e3b3e669-f53f-4070-bbc7-408628639dca","Type":"ContainerStarted","Data":"7f58c7bfce8aae581580c68c8a22793dc4e9a7f59be530b895c045eaca1f78c0"} Nov 02 22:17:14.987896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:14.987869 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" event={"ID":"23ec1893-7330-4af0-91d1-448ba3809d8c","Type":"ContainerStarted","Data":"bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167"} Nov 02 22:17:15.103776 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:15.103725 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" podStartSLOduration=2.103712482 podStartE2EDuration="2.103712482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:15.101815448 +0000 UTC m=+671.366808668" watchObservedRunningTime="2026-11-02 22:17:15.103712482 +0000 UTC m=+671.368705700" Nov 02 22:17:15.755745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:15.755704 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:17:15.756181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:15.755866 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:17:15.756181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:15.755915 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 22:17:15.909135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:15.909088 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:17:16.756089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:16.756052 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:17:16.756504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:16.756111 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 22:17:17.756323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:17.756280 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:17:17.756722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:17.756349 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 22:17:18.395592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.391236 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf"] Nov 02 22:17:18.403634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.403604 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.457220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.457176 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf"] Nov 02 22:17:18.499525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.499485 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8263a4\"" Nov 02 22:17:18.499744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.499563 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8263a4-dockercfg-dp55f\"" Nov 02 22:17:18.505313 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.505288 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkscq\" (UniqueName: \"kubernetes.io/projected/e2cd7261-e112-4be3-936b-7384b02b4889-kube-api-access-jkscq\") pod \"payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.505450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.505352 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cd7261-e112-4be3-936b-7384b02b4889-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.605992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.605947 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkscq\" (UniqueName: \"kubernetes.io/projected/e2cd7261-e112-4be3-936b-7384b02b4889-kube-api-access-jkscq\") pod \"payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.606212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.606051 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cd7261-e112-4be3-936b-7384b02b4889-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.607165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.607135 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cd7261-e112-4be3-936b-7384b02b4889-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.664712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.664637 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkscq\" (UniqueName: \"kubernetes.io/projected/e2cd7261-e112-4be3-936b-7384b02b4889-kube-api-access-jkscq\") pod \"payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.721561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.721525 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:18.756124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.756092 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:17:18.756199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:18.756156 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 22:17:19.117294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:19.117270 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf"] Nov 02 22:17:19.120076 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:17:19.120044 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cd7261_e112_4be3_936b_7384b02b4889.slice/crio-74c5f69137e4788a8b046b976b8d998e68d4b7f432d2fe3e6995b1fe9f384156 WatchSource:0}: Error finding container 74c5f69137e4788a8b046b976b8d998e68d4b7f432d2fe3e6995b1fe9f384156: Status 404 returned error can't find the container with id 74c5f69137e4788a8b046b976b8d998e68d4b7f432d2fe3e6995b1fe9f384156 Nov 02 22:17:19.755828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:19.755789 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:17:19.755994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:19.755846 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 22:17:20.018027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:20.017936 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" event={"ID":"e2cd7261-e112-4be3-936b-7384b02b4889","Type":"ContainerStarted","Data":"c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b"} Nov 02 22:17:20.018027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:20.017990 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" event={"ID":"e2cd7261-e112-4be3-936b-7384b02b4889","Type":"ContainerStarted","Data":"74c5f69137e4788a8b046b976b8d998e68d4b7f432d2fe3e6995b1fe9f384156"} Nov 02 22:17:20.018212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:20.018053 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:20.129209 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:20.129154 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" podStartSLOduration=2.129140788 podStartE2EDuration="2.129140788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:20.127588223 +0000 UTC m=+676.392581436" watchObservedRunningTime="2026-11-02 22:17:20.129140788 +0000 UTC m=+676.394134032" Nov 02 22:17:20.759794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:20.759764 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:17:20.760073 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:20.760050 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:17:20.760754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:20.760736 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:17:24.991031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:24.991002 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:17:25.996839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:25.996804 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:17:26.492692 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:17:26.492653 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:17:28.472799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.472759 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc"] Nov 02 22:17:28.478689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.478664 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.510602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.510541 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blcf7\" (UniqueName: \"kubernetes.io/projected/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-kube-api-access-blcf7\") pod \"payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.510802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.510710 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.529658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.529614 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-98a222\"" Nov 02 22:17:28.576081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.576044 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-98a222-dockercfg-ktc9t\"" Nov 02 22:17:28.578685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.578651 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc"] Nov 02 22:17:28.612656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.612250 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blcf7\" (UniqueName: \"kubernetes.io/projected/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-kube-api-access-blcf7\") pod \"payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.613186 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.613159 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.614327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.614199 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.645366 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.645327 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4"] Nov 02 22:17:28.658641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.658610 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:28.673501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.673464 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blcf7\" (UniqueName: \"kubernetes.io/projected/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-kube-api-access-blcf7\") pod \"payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.710604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.709492 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4"] Nov 02 22:17:28.716101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.716017 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927w7\" (UniqueName: \"kubernetes.io/projected/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-kube-api-access-927w7\") pod \"payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:28.716287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.716134 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:28.794340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.794297 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:28.817246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.817205 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927w7\" (UniqueName: \"kubernetes.io/projected/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-kube-api-access-927w7\") pod \"payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:28.817448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.817344 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:28.818642 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.818509 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:28.877556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.877515 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927w7\" (UniqueName: \"kubernetes.io/projected/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-kube-api-access-927w7\") pod \"payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:28.987148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:28.987104 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:29.173163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:29.173123 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc"] Nov 02 22:17:29.174099 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:17:29.174044 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57460eb1_1c58_44dd_b7a8_b2f9ad12e627.slice/crio-77fdceeeb6dc5444dc8cd1dcb8c589b80cf98e0e69bfdb65c7768df62a662e3c WatchSource:0}: Error finding container 77fdceeeb6dc5444dc8cd1dcb8c589b80cf98e0e69bfdb65c7768df62a662e3c: Status 404 returned error can't find the container with id 77fdceeeb6dc5444dc8cd1dcb8c589b80cf98e0e69bfdb65c7768df62a662e3c Nov 02 22:17:29.365906 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:29.365878 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4"] Nov 02 22:17:29.367847 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:17:29.367811 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4b4ea2_9005_4fcf_9de2_29a19050ca73.slice/crio-a76c89dd8398eb4f2b0e0bea29e7552ee2649ce6a8b14f81ceee63f3758928e0 WatchSource:0}: Error finding container a76c89dd8398eb4f2b0e0bea29e7552ee2649ce6a8b14f81ceee63f3758928e0: Status 404 returned error can't find the container with id a76c89dd8398eb4f2b0e0bea29e7552ee2649ce6a8b14f81ceee63f3758928e0 Nov 02 22:17:30.094962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.094922 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" event={"ID":"5a4b4ea2-9005-4fcf-9de2-29a19050ca73","Type":"ContainerStarted","Data":"7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49"} Nov 02 22:17:30.095454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.094993 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" event={"ID":"5a4b4ea2-9005-4fcf-9de2-29a19050ca73","Type":"ContainerStarted","Data":"a76c89dd8398eb4f2b0e0bea29e7552ee2649ce6a8b14f81ceee63f3758928e0"} Nov 02 22:17:30.095454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.095026 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:30.096715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.096678 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" event={"ID":"57460eb1-1c58-44dd-b7a8-b2f9ad12e627","Type":"ContainerStarted","Data":"b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0"} Nov 02 22:17:30.096875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.096733 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" event={"ID":"57460eb1-1c58-44dd-b7a8-b2f9ad12e627","Type":"ContainerStarted","Data":"77fdceeeb6dc5444dc8cd1dcb8c589b80cf98e0e69bfdb65c7768df62a662e3c"} Nov 02 22:17:30.096875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.096774 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:30.210716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.210654 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" podStartSLOduration=2.210637529 podStartE2EDuration="2.210637529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:30.209799597 +0000 UTC m=+686.474792830" watchObservedRunningTime="2026-11-02 22:17:30.210637529 +0000 UTC m=+686.475630749" Nov 02 22:17:30.321477 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:30.321411 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" podStartSLOduration=2.321390947 podStartE2EDuration="2.321390947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:30.320373219 +0000 UTC m=+686.585366463" watchObservedRunningTime="2026-11-02 22:17:30.321390947 +0000 UTC m=+686.586384168" Nov 02 22:17:31.025918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:31.025889 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:17:32.109024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:32.108974 5109 generic.go:356] "Generic (PLEG): container finished" podID="d7539982-c036-4526-9d1f-bc9324bed23f" containerID="bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d" exitCode=0 Nov 02 22:17:32.109501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:32.109044 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" event={"ID":"d7539982-c036-4526-9d1f-bc9324bed23f","Type":"ContainerDied","Data":"bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d"} Nov 02 22:17:32.109501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:32.109422 5109 scope.go:122] "RemoveContainer" containerID="bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d" Nov 02 22:17:32.110802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:32.110776 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:32.110891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:32.110838 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:32.110891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:32.110869 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:33.116914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:33.116877 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" event={"ID":"d7539982-c036-4526-9d1f-bc9324bed23f","Type":"ContainerStarted","Data":"a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d"} Nov 02 22:17:34.084208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:34.084166 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:17:34.085003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:34.084978 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:17:34.085119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:34.085032 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:17:35.084711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:35.084673 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:17:35.085128 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:35.084728 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:17:36.084121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:36.084084 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:17:36.084298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:36.084145 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:17:37.084172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:37.084127 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:17:37.084629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:37.084198 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:17:37.741593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:37.740682 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:17:38.084108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.084065 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:17:38.084340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.084126 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:17:38.836912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.836867 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9"] Nov 02 22:17:38.842163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.842131 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:38.896782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.896701 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9"] Nov 02 22:17:38.940391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.940363 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-ca85db-dockercfg-md5gm\"" Nov 02 22:17:38.940584 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.940414 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-ca85db\"" Nov 02 22:17:38.996644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.996602 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d566z\" (UniqueName: \"kubernetes.io/projected/d3a5d192-58f0-4d9f-8111-1308242009bd-kube-api-access-d566z\") pod \"payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:38.996858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:38.996777 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d192-58f0-4d9f-8111-1308242009bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:39.089482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.089398 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:17:39.090471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.090445 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:17:39.098173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.098140 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d192-58f0-4d9f-8111-1308242009bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:39.098302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.098282 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d566z\" (UniqueName: \"kubernetes.io/projected/d3a5d192-58f0-4d9f-8111-1308242009bd-kube-api-access-d566z\") pod \"payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:39.098894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.098871 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d192-58f0-4d9f-8111-1308242009bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:39.156524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.156487 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d566z\" (UniqueName: \"kubernetes.io/projected/d3a5d192-58f0-4d9f-8111-1308242009bd-kube-api-access-d566z\") pod \"payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:39.159735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.159708 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:39.533499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:39.533471 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9"] Nov 02 22:17:39.536026 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:17:39.535995 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a5d192_58f0_4d9f_8111_1308242009bd.slice/crio-a75d141f1f59f42125d60f5a829232f82f90ac92d2ad6a46c7fd638e53cc4a4a WatchSource:0}: Error finding container a75d141f1f59f42125d60f5a829232f82f90ac92d2ad6a46c7fd638e53cc4a4a: Status 404 returned error can't find the container with id a75d141f1f59f42125d60f5a829232f82f90ac92d2ad6a46c7fd638e53cc4a4a Nov 02 22:17:40.187944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:40.187899 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" event={"ID":"d3a5d192-58f0-4d9f-8111-1308242009bd","Type":"ContainerStarted","Data":"5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a"} Nov 02 22:17:40.188433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:40.187955 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" event={"ID":"d3a5d192-58f0-4d9f-8111-1308242009bd","Type":"ContainerStarted","Data":"a75d141f1f59f42125d60f5a829232f82f90ac92d2ad6a46c7fd638e53cc4a4a"} Nov 02 22:17:40.188433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:40.188003 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:40.299645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:40.299582 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" podStartSLOduration=2.299548285 podStartE2EDuration="2.299548285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:40.297947201 +0000 UTC m=+696.562940419" watchObservedRunningTime="2026-11-02 22:17:40.299548285 +0000 UTC m=+696.564541506" Nov 02 22:17:41.105303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:41.105271 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:17:41.105520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:41.105341 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:17:51.197186 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:51.197154 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:17:52.274646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:52.274614 5109 generic.go:356] "Generic (PLEG): container finished" podID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerID="877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a" exitCode=0 Nov 02 22:17:52.275127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:52.274651 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" event={"ID":"55d2d9c2-5758-4033-a9b8-26e322a0af11","Type":"ContainerDied","Data":"877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a"} Nov 02 22:17:52.275127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:52.275020 5109 scope.go:122] "RemoveContainer" containerID="877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a" Nov 02 22:17:52.276471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:52.276440 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:52.276626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:52.276525 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:52.276626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:52.276597 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:17:53.282322 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:53.282288 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" event={"ID":"55d2d9c2-5758-4033-a9b8-26e322a0af11","Type":"ContainerStarted","Data":"873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673"} Nov 02 22:17:53.739931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:53.739842 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:17:53.742755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:53.742716 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:17:53.743011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:53.742979 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 22:17:54.744992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:54.744956 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:17:54.745623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:54.745587 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 22:17:54.751344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:54.751317 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:17:55.738040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:55.737999 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:17:55.738803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:55.738354 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 22:17:56.512594 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:17:56.512522 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:17:56.736595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:56.731983 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:17:56.736595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:56.732057 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 22:17:57.733607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:57.733015 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:17:57.733607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:57.733080 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 22:17:58.735784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:58.735748 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:17:58.736954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:17:58.736931 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:18:26.493189 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:18:26.493100 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:18:56.490825 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:18:56.490784 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:19:09.384666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.384632 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn"] Nov 02 22:19:09.388969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.388947 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.466857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.466819 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qtl6\" (UniqueName: \"kubernetes.io/projected/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-kube-api-access-6qtl6\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.467032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.466883 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-home\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.467078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.467051 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-dshm\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.467114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.467101 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tls-certs\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.467200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.467183 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tmp-dir\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.467260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.467248 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-model-cache\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.484956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.484873 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3224ea\"/\"openshift-service-ca.crt\"" Nov 02 22:19:09.485201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.485115 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3224ea\"/\"kube-root-ca.crt\"" Nov 02 22:19:09.485201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.485137 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3224ea\"/\"default-dockercfg-twqdc\"" Nov 02 22:19:09.485201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.485188 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3224ea\"/\"premium-w4-3224ea-kserve-self-signed-certs\"" Nov 02 22:19:09.487676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.487653 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn"] Nov 02 22:19:09.568002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.567961 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qtl6\" (UniqueName: \"kubernetes.io/projected/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-kube-api-access-6qtl6\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568010 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-home\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568064 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-dshm\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568092 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tls-certs\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568140 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tmp-dir\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568201 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-model-cache\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568501 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-home\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568540 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tmp-dir\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.568756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.568729 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-model-cache\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.570547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.570524 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-dshm\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.570750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.570735 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tls-certs\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.626943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.626913 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qtl6\" (UniqueName: \"kubernetes.io/projected/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-kube-api-access-6qtl6\") pod \"premium-w4-3224ea-kserve-84d67555b9-fgmhn\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:09.701880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:09.701777 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:10.109059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:10.109035 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn"] Nov 02 22:19:10.111158 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:19:10.111130 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f43ab2f_5a4c_4759_9212_ae3ab7f7d463.slice/crio-9c16d9bf088965f60bf1f235d809e19c75f685051a193defcfd791886b55be29 WatchSource:0}: Error finding container 9c16d9bf088965f60bf1f235d809e19c75f685051a193defcfd791886b55be29: Status 404 returned error can't find the container with id 9c16d9bf088965f60bf1f235d809e19c75f685051a193defcfd791886b55be29 Nov 02 22:19:10.795895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:10.795859 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" event={"ID":"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463","Type":"ContainerStarted","Data":"655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6"} Nov 02 22:19:10.796317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:10.795904 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" event={"ID":"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463","Type":"ContainerStarted","Data":"9c16d9bf088965f60bf1f235d809e19c75f685051a193defcfd791886b55be29"} Nov 02 22:19:10.907249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:10.907189 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podStartSLOduration=1.907168464 podStartE2EDuration="1.907168464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:10.906343807 +0000 UTC m=+787.171337028" watchObservedRunningTime="2026-11-02 22:19:10.907168464 +0000 UTC m=+787.172161683" Nov 02 22:19:11.702630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:11.702586 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:19.702151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:19.702107 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:19.716732 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:19.716699 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:19.864049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:19.864013 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:19:26.490731 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:19:26.490695 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:19:38.210403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.210359 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp"] Nov 02 22:19:38.213943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.213922 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.263335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.263303 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3224ea-openshift-default-dockercfg-2whsf\"" Nov 02 22:19:38.286956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.286916 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287056 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287131 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5ef48c-b4a6-4145-97aa-222763ba1453-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287150 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287181 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287355 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287388 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287407 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.287437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.287430 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzn2h\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-kube-api-access-rzn2h\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.313917 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.313881 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp"] Nov 02 22:19:38.388476 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388434 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzn2h\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-kube-api-access-rzn2h\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.388713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388498 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.388713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388555 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.388713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388604 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5ef48c-b4a6-4145-97aa-222763ba1453-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.388713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388623 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.388713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388655 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388760 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388875 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.388933 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.389012 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.389111 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.389149 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.389197 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.389459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.389441 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5ef48c-b4a6-4145-97aa-222763ba1453-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.391118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.391099 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.391452 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.391434 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.447328 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.447295 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.447496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.447390 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzn2h\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-kube-api-access-rzn2h\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.560267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.560223 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:38.983286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.983256 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp"] Nov 02 22:19:38.985041 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:19:38.985015 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5ef48c_b4a6_4145_97aa_222763ba1453.slice/crio-753cae0b36b547fe37888ed82c288c7091df582419296f7eca174b0732750460 WatchSource:0}: Error finding container 753cae0b36b547fe37888ed82c288c7091df582419296f7eca174b0732750460: Status 404 returned error can't find the container with id 753cae0b36b547fe37888ed82c288c7091df582419296f7eca174b0732750460 Nov 02 22:19:38.987392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.987359 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:19:38.987467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.987433 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:19:38.987514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:38.987464 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:19:39.992919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:39.992883 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" event={"ID":"9c5ef48c-b4a6-4145-97aa-222763ba1453","Type":"ContainerStarted","Data":"d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e"} Nov 02 22:19:39.993391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:39.992945 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" event={"ID":"9c5ef48c-b4a6-4145-97aa-222763ba1453","Type":"ContainerStarted","Data":"753cae0b36b547fe37888ed82c288c7091df582419296f7eca174b0732750460"} Nov 02 22:19:40.207033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:40.206969 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" podStartSLOduration=2.206949158 podStartE2EDuration="2.206949158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:40.160056292 +0000 UTC m=+816.425049524" watchObservedRunningTime="2026-11-02 22:19:40.206949158 +0000 UTC m=+816.471942381" Nov 02 22:19:40.561479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:40.561433 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:40.562803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:40.562779 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:19:40.562876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:40.562832 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 22:19:41.563654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:41.560968 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:19:41.563654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:41.561031 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 22:19:42.561290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:42.561250 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:19:42.561489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:42.561322 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 22:19:43.561245 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:43.561199 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:19:43.561709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:43.561265 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 22:19:44.561340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:44.561294 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:19:44.561778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:44.561373 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 22:19:45.795809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:45.795763 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:45.796318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:45.796133 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:45.797031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:19:45.797013 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:19:56.490963 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:19:56.490905 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:20:03.796759 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.796720 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9"] Nov 02 22:20:03.801053 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.801031 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:03.853369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.853331 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:03.856843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.856809 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9"] Nov 02 22:20:03.899783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.899746 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:20:03.903056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.903028 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:03.903228 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.903082 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:03.903228 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:03.903148 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qcm\" (UniqueName: \"kubernetes.io/projected/e490a7bf-cae0-4d82-81a8-b80327649fb3-kube-api-access-88qcm\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.004587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.004538 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.004763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.004630 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qcm\" (UniqueName: \"kubernetes.io/projected/e490a7bf-cae0-4d82-81a8-b80327649fb3-kube-api-access-88qcm\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.004763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.004739 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.005326 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.005299 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.007239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.007220 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.064817 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.064741 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qcm\" (UniqueName: \"kubernetes.io/projected/e490a7bf-cae0-4d82-81a8-b80327649fb3-kube-api-access-88qcm\") pod \"maas-api-revoke-sub-9b69049a-w4ns9\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.114875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.114840 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:04.484263 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:04.484234 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9"] Nov 02 22:20:04.486130 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:04.486095 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode490a7bf_cae0_4d82_81a8_b80327649fb3.slice/crio-845d1a1cdc9b7593e6e885d3cd8fa61710c93b3081fd675435acac7e6e2585c3 WatchSource:0}: Error finding container 845d1a1cdc9b7593e6e885d3cd8fa61710c93b3081fd675435acac7e6e2585c3: Status 404 returned error can't find the container with id 845d1a1cdc9b7593e6e885d3cd8fa61710c93b3081fd675435acac7e6e2585c3 Nov 02 22:20:05.136829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:05.136797 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" event={"ID":"e490a7bf-cae0-4d82-81a8-b80327649fb3","Type":"ContainerStarted","Data":"1450ec703306243a5019bf884785d17ea9cac6b4d88be91998abe4173c505aab"} Nov 02 22:20:05.137223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:05.136856 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" event={"ID":"e490a7bf-cae0-4d82-81a8-b80327649fb3","Type":"ContainerStarted","Data":"845d1a1cdc9b7593e6e885d3cd8fa61710c93b3081fd675435acac7e6e2585c3"} Nov 02 22:20:05.249077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:05.249026 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" podStartSLOduration=2.249011831 podStartE2EDuration="2.249011831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:05.246790735 +0000 UTC m=+841.511783957" watchObservedRunningTime="2026-11-02 22:20:05.249011831 +0000 UTC m=+841.514005049" Nov 02 22:20:06.143537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:06.143499 5109 generic.go:356] "Generic (PLEG): container finished" podID="e490a7bf-cae0-4d82-81a8-b80327649fb3" containerID="1450ec703306243a5019bf884785d17ea9cac6b4d88be91998abe4173c505aab" exitCode=0 Nov 02 22:20:06.143971 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:06.143612 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" event={"ID":"e490a7bf-cae0-4d82-81a8-b80327649fb3","Type":"ContainerDied","Data":"1450ec703306243a5019bf884785d17ea9cac6b4d88be91998abe4173c505aab"} Nov 02 22:20:07.294410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.294379 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:07.443079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.442990 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-openshift-ca\") pod \"e490a7bf-cae0-4d82-81a8-b80327649fb3\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " Nov 02 22:20:07.443079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.443047 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-xks-ca\") pod \"e490a7bf-cae0-4d82-81a8-b80327649fb3\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " Nov 02 22:20:07.443281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.443102 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e490a7bf-cae0-4d82-81a8-b80327649fb3-kube-api-access-88qcm\" (UniqueName: \"kubernetes.io/projected/e490a7bf-cae0-4d82-81a8-b80327649fb3-kube-api-access-88qcm\") pod \"e490a7bf-cae0-4d82-81a8-b80327649fb3\" (UID: \"e490a7bf-cae0-4d82-81a8-b80327649fb3\") " Nov 02 22:20:07.443384 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.443361 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-openshift-ca" pod "e490a7bf-cae0-4d82-81a8-b80327649fb3" (UID: "e490a7bf-cae0-4d82-81a8-b80327649fb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:07.445383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.445351 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e490a7bf-cae0-4d82-81a8-b80327649fb3-kube-api-access-88qcm" pod "e490a7bf-cae0-4d82-81a8-b80327649fb3" (UID: "e490a7bf-cae0-4d82-81a8-b80327649fb3"). InnerVolumeSpecName "kube-api-access-88qcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:07.445383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.445351 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-xks-ca" pod "e490a7bf-cae0-4d82-81a8-b80327649fb3" (UID: "e490a7bf-cae0-4d82-81a8-b80327649fb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:07.544308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.544264 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:07.544308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.544300 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qcm\" (UniqueName: \"kubernetes.io/projected/e490a7bf-cae0-4d82-81a8-b80327649fb3-kube-api-access-88qcm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:07.544308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:07.544311 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e490a7bf-cae0-4d82-81a8-b80327649fb3-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:08.155637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:08.155535 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" event={"ID":"e490a7bf-cae0-4d82-81a8-b80327649fb3","Type":"ContainerDied","Data":"845d1a1cdc9b7593e6e885d3cd8fa61710c93b3081fd675435acac7e6e2585c3"} Nov 02 22:20:08.155637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:08.155593 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9" Nov 02 22:20:08.155637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:08.155619 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="845d1a1cdc9b7593e6e885d3cd8fa61710c93b3081fd675435acac7e6e2585c3" Nov 02 22:20:09.521541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.521505 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72"] Nov 02 22:20:09.522138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.522113 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e490a7bf-cae0-4d82-81a8-b80327649fb3" containerName="revoke-keys" Nov 02 22:20:09.522138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.522134 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e490a7bf-cae0-4d82-81a8-b80327649fb3" containerName="revoke-keys" Nov 02 22:20:09.522302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.522231 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e490a7bf-cae0-4d82-81a8-b80327649fb3" containerName="revoke-keys" Nov 02 22:20:09.525999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.525976 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.586007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.585969 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"openshift-service-ca.crt\"" Nov 02 22:20:09.621929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.621866 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"kube-root-ca.crt\"" Nov 02 22:20:09.622111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.621960 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"default-dockercfg-9cwrj\"" Nov 02 22:20:09.622111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.621957 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"premium-w3-98a222-kserve-self-signed-certs\"" Nov 02 22:20:09.624421 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.624397 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72"] Nov 02 22:20:09.670962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.670918 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7zt\" (UniqueName: \"kubernetes.io/projected/f5155c18-83ec-4923-9855-dd0c6f293187-kube-api-access-9k7zt\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.671251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.671217 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-dshm\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.671375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.671276 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-tmp-dir\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.671375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.671327 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-model-cache\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.671375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.671352 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5155c18-83ec-4923-9855-dd0c6f293187-tls-certs\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.671535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.671433 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-home\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.772383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772272 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7zt\" (UniqueName: \"kubernetes.io/projected/f5155c18-83ec-4923-9855-dd0c6f293187-kube-api-access-9k7zt\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.772383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772317 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-dshm\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.772718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772457 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-tmp-dir\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.772718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772624 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-model-cache\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.772718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772663 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5155c18-83ec-4923-9855-dd0c6f293187-tls-certs\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.772882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772764 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-home\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.772882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772852 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-tmp-dir\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.773017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.772990 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-model-cache\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.773218 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.773199 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-home\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.774820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.774795 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-dshm\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.775277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.775257 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5155c18-83ec-4923-9855-dd0c6f293187-tls-certs\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.832122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.832086 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7zt\" (UniqueName: \"kubernetes.io/projected/f5155c18-83ec-4923-9855-dd0c6f293187-kube-api-access-9k7zt\") pod \"premium-w3-98a222-kserve-84c8ccf58f-9ft72\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:09.839029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:09.838992 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:10.254089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:10.254057 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72"] Nov 02 22:20:10.255702 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:10.255674 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5155c18_83ec_4923_9855_dd0c6f293187.slice/crio-f0afed405481439d40d07127ea96793a28fc6f3dce4b8f791a04ec0f630fb974 WatchSource:0}: Error finding container f0afed405481439d40d07127ea96793a28fc6f3dce4b8f791a04ec0f630fb974: Status 404 returned error can't find the container with id f0afed405481439d40d07127ea96793a28fc6f3dce4b8f791a04ec0f630fb974 Nov 02 22:20:11.175127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:11.175093 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" event={"ID":"f5155c18-83ec-4923-9855-dd0c6f293187","Type":"ContainerStarted","Data":"7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f"} Nov 02 22:20:11.175127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:11.175140 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" event={"ID":"f5155c18-83ec-4923-9855-dd0c6f293187","Type":"ContainerStarted","Data":"f0afed405481439d40d07127ea96793a28fc6f3dce4b8f791a04ec0f630fb974"} Nov 02 22:20:11.286320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:11.286262 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podStartSLOduration=2.286246954 podStartE2EDuration="2.286246954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:11.28547079 +0000 UTC m=+847.550464010" watchObservedRunningTime="2026-11-02 22:20:11.286246954 +0000 UTC m=+847.551240172" Nov 02 22:20:11.839546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:11.839501 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:12.575868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.575825 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd"] Nov 02 22:20:12.579844 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.579823 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.629560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.629529 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:12.676411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.676382 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:20:12.679530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.679505 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd"] Nov 02 22:20:12.727114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.727077 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9fs\" (UniqueName: \"kubernetes.io/projected/04d4af33-9f66-49f1-9bdd-f3500fad3569-kube-api-access-bw9fs\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.727311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.727139 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.727311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.727264 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.828720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.828623 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.828720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.828697 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.828954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.828794 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9fs\" (UniqueName: \"kubernetes.io/projected/04d4af33-9f66-49f1-9bdd-f3500fad3569-kube-api-access-bw9fs\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.829429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.829401 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.831193 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.831165 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.887142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.887103 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9fs\" (UniqueName: \"kubernetes.io/projected/04d4af33-9f66-49f1-9bdd-f3500fad3569-kube-api-access-bw9fs\") pod \"maas-api-revoke-sub-bc1530f7-fbkzd\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:12.893809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:12.893779 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:13.261746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:13.261707 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd"] Nov 02 22:20:13.263358 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:13.263331 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d4af33_9f66_49f1_9bdd_f3500fad3569.slice/crio-7d00358b6f8b1ace70a095633c626a9996859b54a38e2ad5375cffca64cfd060 WatchSource:0}: Error finding container 7d00358b6f8b1ace70a095633c626a9996859b54a38e2ad5375cffca64cfd060: Status 404 returned error can't find the container with id 7d00358b6f8b1ace70a095633c626a9996859b54a38e2ad5375cffca64cfd060 Nov 02 22:20:14.193931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:14.193888 5109 generic.go:356] "Generic (PLEG): container finished" podID="04d4af33-9f66-49f1-9bdd-f3500fad3569" containerID="d77b91bf8d25ba9c7973071a885cead03e5168639a44a31755cc4792ba273d32" exitCode=0 Nov 02 22:20:14.194309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:14.193937 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" event={"ID":"04d4af33-9f66-49f1-9bdd-f3500fad3569","Type":"ContainerDied","Data":"d77b91bf8d25ba9c7973071a885cead03e5168639a44a31755cc4792ba273d32"} Nov 02 22:20:14.194309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:14.194024 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" event={"ID":"04d4af33-9f66-49f1-9bdd-f3500fad3569","Type":"ContainerStarted","Data":"7d00358b6f8b1ace70a095633c626a9996859b54a38e2ad5375cffca64cfd060"} Nov 02 22:20:15.345768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.345739 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:15.359661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.359626 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-xks-ca\") pod \"04d4af33-9f66-49f1-9bdd-f3500fad3569\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " Nov 02 22:20:15.359810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.359668 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d4af33-9f66-49f1-9bdd-f3500fad3569-kube-api-access-bw9fs\" (UniqueName: \"kubernetes.io/projected/04d4af33-9f66-49f1-9bdd-f3500fad3569-kube-api-access-bw9fs\") pod \"04d4af33-9f66-49f1-9bdd-f3500fad3569\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " Nov 02 22:20:15.359810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.359730 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-openshift-ca\") pod \"04d4af33-9f66-49f1-9bdd-f3500fad3569\" (UID: \"04d4af33-9f66-49f1-9bdd-f3500fad3569\") " Nov 02 22:20:15.360166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.360139 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-openshift-ca" pod "04d4af33-9f66-49f1-9bdd-f3500fad3569" (UID: "04d4af33-9f66-49f1-9bdd-f3500fad3569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:15.361808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.361788 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-xks-ca" pod "04d4af33-9f66-49f1-9bdd-f3500fad3569" (UID: "04d4af33-9f66-49f1-9bdd-f3500fad3569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:15.362014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.361998 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d4af33-9f66-49f1-9bdd-f3500fad3569-kube-api-access-bw9fs" pod "04d4af33-9f66-49f1-9bdd-f3500fad3569" (UID: "04d4af33-9f66-49f1-9bdd-f3500fad3569"). InnerVolumeSpecName "kube-api-access-bw9fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:15.460858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.460781 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:15.460858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.460810 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d4af33-9f66-49f1-9bdd-f3500fad3569-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:15.460858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:15.460827 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9fs\" (UniqueName: \"kubernetes.io/projected/04d4af33-9f66-49f1-9bdd-f3500fad3569-kube-api-access-bw9fs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:16.207888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:16.207812 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" Nov 02 22:20:16.208043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:16.207819 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd" event={"ID":"04d4af33-9f66-49f1-9bdd-f3500fad3569","Type":"ContainerDied","Data":"7d00358b6f8b1ace70a095633c626a9996859b54a38e2ad5375cffca64cfd060"} Nov 02 22:20:16.208043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:16.207936 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d00358b6f8b1ace70a095633c626a9996859b54a38e2ad5375cffca64cfd060" Nov 02 22:20:19.245759 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.245726 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv"] Nov 02 22:20:19.246290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.246275 5109 cpu_manager.go:414] "containerMap: removing container" podUID="04d4af33-9f66-49f1-9bdd-f3500fad3569" containerName="revoke-keys" Nov 02 22:20:19.246345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.246291 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d4af33-9f66-49f1-9bdd-f3500fad3569" containerName="revoke-keys" Nov 02 22:20:19.246431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.246421 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d4af33-9f66-49f1-9bdd-f3500fad3569" containerName="revoke-keys" Nov 02 22:20:19.249622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.249603 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.298839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.298797 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:19.316846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.316805 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6sc\" (UniqueName: \"kubernetes.io/projected/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-kube-api-access-fh6sc\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.317036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.316929 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.317036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.317001 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.345590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.345544 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:20:19.348050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.348024 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv"] Nov 02 22:20:19.418645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.418606 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6sc\" (UniqueName: \"kubernetes.io/projected/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-kube-api-access-fh6sc\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.418826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.418663 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.418826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.418714 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.419464 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.419445 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.421197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.421175 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.477397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.477362 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6sc\" (UniqueName: \"kubernetes.io/projected/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-kube-api-access-fh6sc\") pod \"maas-api-revoke-sub-1f341310-cs6bv\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.563780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.563730 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:19.839839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.839753 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:19.852697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.852665 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:19.935316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:19.935285 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv"] Nov 02 22:20:19.937068 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:19.937039 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded01dc6a_51cc_4d4a_87d6_5b3a0e71aa50.slice/crio-33c6cf78fa35058c06a84a00518c26f810ee72b2c70e20dc2204b62d9a57fb44 WatchSource:0}: Error finding container 33c6cf78fa35058c06a84a00518c26f810ee72b2c70e20dc2204b62d9a57fb44: Status 404 returned error can't find the container with id 33c6cf78fa35058c06a84a00518c26f810ee72b2c70e20dc2204b62d9a57fb44 Nov 02 22:20:20.233493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:20.233448 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" event={"ID":"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50","Type":"ContainerStarted","Data":"33c6cf78fa35058c06a84a00518c26f810ee72b2c70e20dc2204b62d9a57fb44"} Nov 02 22:20:20.245425 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:20.245394 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:20:21.243032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:21.242939 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" containerID="fba6a181555709f27295d138c1d6c8669b9dd236d1e37992e0b0f5c5a67bf0be" exitCode=0 Nov 02 22:20:21.243452 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:21.243025 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" event={"ID":"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50","Type":"ContainerDied","Data":"fba6a181555709f27295d138c1d6c8669b9dd236d1e37992e0b0f5c5a67bf0be"} Nov 02 22:20:22.401514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.401485 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:22.451682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.451644 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-openshift-ca\") pod \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " Nov 02 22:20:22.451881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.451714 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-xks-ca\") pod \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " Nov 02 22:20:22.451881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.451806 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-kube-api-access-fh6sc\" (UniqueName: \"kubernetes.io/projected/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-kube-api-access-fh6sc\") pod \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\" (UID: \"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50\") " Nov 02 22:20:22.452029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.452007 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-openshift-ca" pod "ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" (UID: "ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:22.452223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.452208 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.454007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.453977 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-xks-ca" pod "ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" (UID: "ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:22.454007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.453984 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-kube-api-access-fh6sc" pod "ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" (UID: "ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50"). InnerVolumeSpecName "kube-api-access-fh6sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:22.553523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.553484 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.553523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:22.553519 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6sc\" (UniqueName: \"kubernetes.io/projected/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50-kube-api-access-fh6sc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.255918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:23.255888 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" Nov 02 22:20:23.255918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:23.255902 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv" event={"ID":"ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50","Type":"ContainerDied","Data":"33c6cf78fa35058c06a84a00518c26f810ee72b2c70e20dc2204b62d9a57fb44"} Nov 02 22:20:23.256682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:23.255954 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c6cf78fa35058c06a84a00518c26f810ee72b2c70e20dc2204b62d9a57fb44" Nov 02 22:20:26.492639 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:20:26.492600 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:20:38.349312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.349271 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5"] Nov 02 22:20:38.350094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.350071 5109 cpu_manager.go:414] "containerMap: removing container" podUID="ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" containerName="revoke-keys" Nov 02 22:20:38.350190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.350094 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" containerName="revoke-keys" Nov 02 22:20:38.350276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.350262 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" containerName="revoke-keys" Nov 02 22:20:38.354057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.354034 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.439030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.438990 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.439238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.439041 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.439238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.439180 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7v98\" (UniqueName: \"kubernetes.io/projected/5f9e3866-53e0-4efe-949b-9e3fea7c7555-kube-api-access-q7v98\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.452808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.452774 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:38.453072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.453050 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:20:38.455529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.455504 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5"] Nov 02 22:20:38.540370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.540331 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7v98\" (UniqueName: \"kubernetes.io/projected/5f9e3866-53e0-4efe-949b-9e3fea7c7555-kube-api-access-q7v98\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.540637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.540470 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.540637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.540495 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.541146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.541125 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.543022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.542997 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.599472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.599381 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7v98\" (UniqueName: \"kubernetes.io/projected/5f9e3866-53e0-4efe-949b-9e3fea7c7555-kube-api-access-q7v98\") pod \"maas-api-revoke-sub-8ba68aa3-n4rv5\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:38.668321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:38.668284 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:39.043591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.043548 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5"] Nov 02 22:20:39.046120 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:39.046086 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9e3866_53e0_4efe_949b_9e3fea7c7555.slice/crio-3a905471df1ff649b15c51b03c773d29efc455bf2adbee7ad64be467ee3e4804 WatchSource:0}: Error finding container 3a905471df1ff649b15c51b03c773d29efc455bf2adbee7ad64be467ee3e4804: Status 404 returned error can't find the container with id 3a905471df1ff649b15c51b03c773d29efc455bf2adbee7ad64be467ee3e4804 Nov 02 22:20:39.350750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.350715 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" event={"ID":"5f9e3866-53e0-4efe-949b-9e3fea7c7555","Type":"ContainerStarted","Data":"7dee7fd063bd7a79eba26000842dedc5119fbf30ff679bbe0a515a3a3d71eb7d"} Nov 02 22:20:39.351198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.350767 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" event={"ID":"5f9e3866-53e0-4efe-949b-9e3fea7c7555","Type":"ContainerStarted","Data":"3a905471df1ff649b15c51b03c773d29efc455bf2adbee7ad64be467ee3e4804"} Nov 02 22:20:39.463091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.463031 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" podStartSLOduration=1.463018292 podStartE2EDuration="1.463018292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:39.460354741 +0000 UTC m=+875.725347972" watchObservedRunningTime="2026-11-02 22:20:39.463018292 +0000 UTC m=+875.728011510" Nov 02 22:20:39.537020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.536988 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j"] Nov 02 22:20:39.541468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.541442 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.596671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.596557 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j"] Nov 02 22:20:39.653907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.653873 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.654115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.653936 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n27g\" (UniqueName: \"kubernetes.io/projected/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-kube-api-access-4n27g\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.654115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.654081 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.755228 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.755180 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.755441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.755377 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.755517 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.755444 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n27g\" (UniqueName: \"kubernetes.io/projected/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-kube-api-access-4n27g\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.755906 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.755883 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.757988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.757956 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.815780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.815748 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n27g\" (UniqueName: \"kubernetes.io/projected/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-kube-api-access-4n27g\") pod \"maas-api-revoke-sub-4f0c0451-zzq7j\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:39.865516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:39.865428 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:40.238486 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:40.238449 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j"] Nov 02 22:20:40.241151 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:40.241114 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1868ed_4357_43a6_8c81_7ad7fdb356f4.slice/crio-df555e61ff27bab36e75e56b732c57b56f339f256f7ff4b4c3232855f902441e WatchSource:0}: Error finding container df555e61ff27bab36e75e56b732c57b56f339f256f7ff4b4c3232855f902441e: Status 404 returned error can't find the container with id df555e61ff27bab36e75e56b732c57b56f339f256f7ff4b4c3232855f902441e Nov 02 22:20:40.358535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:40.358501 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" event={"ID":"8c1868ed-4357-43a6-8c81-7ad7fdb356f4","Type":"ContainerStarted","Data":"df555e61ff27bab36e75e56b732c57b56f339f256f7ff4b4c3232855f902441e"} Nov 02 22:20:40.360195 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:40.360153 5109 generic.go:356] "Generic (PLEG): container finished" podID="5f9e3866-53e0-4efe-949b-9e3fea7c7555" containerID="7dee7fd063bd7a79eba26000842dedc5119fbf30ff679bbe0a515a3a3d71eb7d" exitCode=0 Nov 02 22:20:40.360340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:40.360207 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" event={"ID":"5f9e3866-53e0-4efe-949b-9e3fea7c7555","Type":"ContainerDied","Data":"7dee7fd063bd7a79eba26000842dedc5119fbf30ff679bbe0a515a3a3d71eb7d"} Nov 02 22:20:41.367260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.367169 5109 generic.go:356] "Generic (PLEG): container finished" podID="8c1868ed-4357-43a6-8c81-7ad7fdb356f4" containerID="b6a29612d67ee0a3e6306239b619b661fe8eccd3b7718b7ca98eb608723a6abb" exitCode=0 Nov 02 22:20:41.367703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.367270 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" event={"ID":"8c1868ed-4357-43a6-8c81-7ad7fdb356f4","Type":"ContainerDied","Data":"b6a29612d67ee0a3e6306239b619b661fe8eccd3b7718b7ca98eb608723a6abb"} Nov 02 22:20:41.504592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.504534 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:41.678217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.678129 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-openshift-ca\") pod \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " Nov 02 22:20:41.678217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.678183 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9e3866-53e0-4efe-949b-9e3fea7c7555-kube-api-access-q7v98\" (UniqueName: \"kubernetes.io/projected/5f9e3866-53e0-4efe-949b-9e3fea7c7555-kube-api-access-q7v98\") pod \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " Nov 02 22:20:41.678415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.678245 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-xks-ca\") pod \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\" (UID: \"5f9e3866-53e0-4efe-949b-9e3fea7c7555\") " Nov 02 22:20:41.678485 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.678463 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-openshift-ca" pod "5f9e3866-53e0-4efe-949b-9e3fea7c7555" (UID: "5f9e3866-53e0-4efe-949b-9e3fea7c7555"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:41.678697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.678679 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:41.680515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.680484 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-xks-ca" pod "5f9e3866-53e0-4efe-949b-9e3fea7c7555" (UID: "5f9e3866-53e0-4efe-949b-9e3fea7c7555"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:41.680653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.680561 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9e3866-53e0-4efe-949b-9e3fea7c7555-kube-api-access-q7v98" pod "5f9e3866-53e0-4efe-949b-9e3fea7c7555" (UID: "5f9e3866-53e0-4efe-949b-9e3fea7c7555"). InnerVolumeSpecName "kube-api-access-q7v98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:41.780010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.779965 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7v98\" (UniqueName: \"kubernetes.io/projected/5f9e3866-53e0-4efe-949b-9e3fea7c7555-kube-api-access-q7v98\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:41.780010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:41.780000 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f9e3866-53e0-4efe-949b-9e3fea7c7555-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:42.374951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.374922 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" Nov 02 22:20:42.375379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.374960 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5" event={"ID":"5f9e3866-53e0-4efe-949b-9e3fea7c7555","Type":"ContainerDied","Data":"3a905471df1ff649b15c51b03c773d29efc455bf2adbee7ad64be467ee3e4804"} Nov 02 22:20:42.375379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.375012 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a905471df1ff649b15c51b03c773d29efc455bf2adbee7ad64be467ee3e4804" Nov 02 22:20:42.524388 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.524356 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:42.688998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.688908 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-xks-ca\") pod \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " Nov 02 22:20:42.688998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.688946 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-openshift-ca\") pod \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " Nov 02 22:20:42.689194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.689038 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-kube-api-access-4n27g\" (UniqueName: \"kubernetes.io/projected/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-kube-api-access-4n27g\") pod \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\" (UID: \"8c1868ed-4357-43a6-8c81-7ad7fdb356f4\") " Nov 02 22:20:42.689415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.689383 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-openshift-ca" pod "8c1868ed-4357-43a6-8c81-7ad7fdb356f4" (UID: "8c1868ed-4357-43a6-8c81-7ad7fdb356f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:42.689537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.689526 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:42.691412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.691386 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-xks-ca" pod "8c1868ed-4357-43a6-8c81-7ad7fdb356f4" (UID: "8c1868ed-4357-43a6-8c81-7ad7fdb356f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:42.691412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.691400 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-kube-api-access-4n27g" pod "8c1868ed-4357-43a6-8c81-7ad7fdb356f4" (UID: "8c1868ed-4357-43a6-8c81-7ad7fdb356f4"). InnerVolumeSpecName "kube-api-access-4n27g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:42.790281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.790240 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:42.790281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:42.790283 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n27g\" (UniqueName: \"kubernetes.io/projected/8c1868ed-4357-43a6-8c81-7ad7fdb356f4-kube-api-access-4n27g\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:43.382901 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:43.382871 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" Nov 02 22:20:43.383286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:43.382908 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j" event={"ID":"8c1868ed-4357-43a6-8c81-7ad7fdb356f4","Type":"ContainerDied","Data":"df555e61ff27bab36e75e56b732c57b56f339f256f7ff4b4c3232855f902441e"} Nov 02 22:20:43.383286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:43.382956 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df555e61ff27bab36e75e56b732c57b56f339f256f7ff4b4c3232855f902441e" Nov 02 22:20:47.674465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.674429 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb"] Nov 02 22:20:47.675132 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.675113 5109 cpu_manager.go:414] "containerMap: removing container" podUID="5f9e3866-53e0-4efe-949b-9e3fea7c7555" containerName="revoke-keys" Nov 02 22:20:47.675173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.675134 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9e3866-53e0-4efe-949b-9e3fea7c7555" containerName="revoke-keys" Nov 02 22:20:47.675225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.675175 5109 cpu_manager.go:414] "containerMap: removing container" podUID="8c1868ed-4357-43a6-8c81-7ad7fdb356f4" containerName="revoke-keys" Nov 02 22:20:47.675225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.675181 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1868ed-4357-43a6-8c81-7ad7fdb356f4" containerName="revoke-keys" Nov 02 22:20:47.675294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.675270 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9e3866-53e0-4efe-949b-9e3fea7c7555" containerName="revoke-keys" Nov 02 22:20:47.675294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.675286 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1868ed-4357-43a6-8c81-7ad7fdb356f4" containerName="revoke-keys" Nov 02 22:20:47.678747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.678730 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.746984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.746953 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.747137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.747004 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fn5x\" (UniqueName: \"kubernetes.io/projected/45ed30e3-6b87-4249-9716-c7524070e155-kube-api-access-9fn5x\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.747137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.747086 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.776485 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.776444 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:47.776770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.776752 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:20:47.779236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.779211 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb"] Nov 02 22:20:47.848438 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.848392 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.848671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.848454 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fn5x\" (UniqueName: \"kubernetes.io/projected/45ed30e3-6b87-4249-9716-c7524070e155-kube-api-access-9fn5x\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.848671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.848502 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.849092 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.849071 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.850992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.850964 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.907105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.907074 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fn5x\" (UniqueName: \"kubernetes.io/projected/45ed30e3-6b87-4249-9716-c7524070e155-kube-api-access-9fn5x\") pod \"maas-api-revoke-sub-410664e3-x5pkb\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:47.992090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:47.992005 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:48.362783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:48.362752 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb"] Nov 02 22:20:48.364001 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:48.363971 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ed30e3_6b87_4249_9716_c7524070e155.slice/crio-ca87c3157135a1887830b8f794b74f2eb3f53ad742ea715c3a0b76537e82c3c1 WatchSource:0}: Error finding container ca87c3157135a1887830b8f794b74f2eb3f53ad742ea715c3a0b76537e82c3c1: Status 404 returned error can't find the container with id ca87c3157135a1887830b8f794b74f2eb3f53ad742ea715c3a0b76537e82c3c1 Nov 02 22:20:48.414622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:48.414562 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" event={"ID":"45ed30e3-6b87-4249-9716-c7524070e155","Type":"ContainerStarted","Data":"ca87c3157135a1887830b8f794b74f2eb3f53ad742ea715c3a0b76537e82c3c1"} Nov 02 22:20:49.421880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:49.421842 5109 generic.go:356] "Generic (PLEG): container finished" podID="45ed30e3-6b87-4249-9716-c7524070e155" containerID="975bb4fce3331e42958c6f50f3c20e0070657b809578df27edfd578c2c739c41" exitCode=0 Nov 02 22:20:49.422266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:49.421926 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" event={"ID":"45ed30e3-6b87-4249-9716-c7524070e155","Type":"ContainerDied","Data":"975bb4fce3331e42958c6f50f3c20e0070657b809578df27edfd578c2c739c41"} Nov 02 22:20:50.307143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.307106 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f"] Nov 02 22:20:50.311022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.311001 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.371864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.371826 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f"] Nov 02 22:20:50.377894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.377859 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwll\" (UniqueName: \"kubernetes.io/projected/1d2fbc73-440a-4209-8eec-a256c1310417-kube-api-access-dxwll\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.378058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.377932 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.378122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.378053 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.479675 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.479630 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwll\" (UniqueName: \"kubernetes.io/projected/1d2fbc73-440a-4209-8eec-a256c1310417-kube-api-access-dxwll\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.480144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.479733 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.480144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.479829 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.480532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.480506 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.482449 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.482427 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.538130 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.538089 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwll\" (UniqueName: \"kubernetes.io/projected/1d2fbc73-440a-4209-8eec-a256c1310417-kube-api-access-dxwll\") pod \"maas-api-revoke-sub-cd3d5118-mns4f\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.596579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.596537 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:50.624553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.624515 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:50.684705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.682944 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-openshift-ca\") pod \"45ed30e3-6b87-4249-9716-c7524070e155\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " Nov 02 22:20:50.684705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.682981 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-xks-ca\") pod \"45ed30e3-6b87-4249-9716-c7524070e155\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " Nov 02 22:20:50.684705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.683058 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ed30e3-6b87-4249-9716-c7524070e155-kube-api-access-9fn5x\" (UniqueName: \"kubernetes.io/projected/45ed30e3-6b87-4249-9716-c7524070e155-kube-api-access-9fn5x\") pod \"45ed30e3-6b87-4249-9716-c7524070e155\" (UID: \"45ed30e3-6b87-4249-9716-c7524070e155\") " Nov 02 22:20:50.684705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.683362 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-openshift-ca" pod "45ed30e3-6b87-4249-9716-c7524070e155" (UID: "45ed30e3-6b87-4249-9716-c7524070e155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:50.684705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.683464 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:50.688461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.688434 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ed30e3-6b87-4249-9716-c7524070e155-kube-api-access-9fn5x" pod "45ed30e3-6b87-4249-9716-c7524070e155" (UID: "45ed30e3-6b87-4249-9716-c7524070e155"). InnerVolumeSpecName "kube-api-access-9fn5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:50.690198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.690171 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-xks-ca" pod "45ed30e3-6b87-4249-9716-c7524070e155" (UID: "45ed30e3-6b87-4249-9716-c7524070e155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:50.784859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.784821 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ed30e3-6b87-4249-9716-c7524070e155-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:50.784859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.784861 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fn5x\" (UniqueName: \"kubernetes.io/projected/45ed30e3-6b87-4249-9716-c7524070e155-kube-api-access-9fn5x\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:50.994108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:50.994080 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f"] Nov 02 22:20:50.995344 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:50.995313 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2fbc73_440a_4209_8eec_a256c1310417.slice/crio-ef0db662193022e8e17fb8da0141e41948d0a80d156a349f402b7f3ebfae08dd WatchSource:0}: Error finding container ef0db662193022e8e17fb8da0141e41948d0a80d156a349f402b7f3ebfae08dd: Status 404 returned error can't find the container with id ef0db662193022e8e17fb8da0141e41948d0a80d156a349f402b7f3ebfae08dd Nov 02 22:20:51.435650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:51.435612 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" Nov 02 22:20:51.435846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:51.435608 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb" event={"ID":"45ed30e3-6b87-4249-9716-c7524070e155","Type":"ContainerDied","Data":"ca87c3157135a1887830b8f794b74f2eb3f53ad742ea715c3a0b76537e82c3c1"} Nov 02 22:20:51.435846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:51.435775 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca87c3157135a1887830b8f794b74f2eb3f53ad742ea715c3a0b76537e82c3c1" Nov 02 22:20:51.437446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:51.437412 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" event={"ID":"1d2fbc73-440a-4209-8eec-a256c1310417","Type":"ContainerStarted","Data":"fff62f324bb4d1ce38a8bc38d6015b6a9098951b8031e7fdf209a830e9191391"} Nov 02 22:20:51.437596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:51.437462 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" event={"ID":"1d2fbc73-440a-4209-8eec-a256c1310417","Type":"ContainerStarted","Data":"ef0db662193022e8e17fb8da0141e41948d0a80d156a349f402b7f3ebfae08dd"} Nov 02 22:20:51.549168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:51.549118 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" podStartSLOduration=1.5491039130000002 podStartE2EDuration="1.549103913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:51.546440593 +0000 UTC m=+887.811433815" watchObservedRunningTime="2026-11-02 22:20:51.549103913 +0000 UTC m=+887.814097132" Nov 02 22:20:52.445309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:52.445276 5109 generic.go:356] "Generic (PLEG): container finished" podID="1d2fbc73-440a-4209-8eec-a256c1310417" containerID="fff62f324bb4d1ce38a8bc38d6015b6a9098951b8031e7fdf209a830e9191391" exitCode=0 Nov 02 22:20:52.445495 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:52.445315 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" event={"ID":"1d2fbc73-440a-4209-8eec-a256c1310417","Type":"ContainerDied","Data":"fff62f324bb4d1ce38a8bc38d6015b6a9098951b8031e7fdf209a830e9191391"} Nov 02 22:20:53.076311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.076275 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66"] Nov 02 22:20:53.076890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.076867 5109 cpu_manager.go:414] "containerMap: removing container" podUID="45ed30e3-6b87-4249-9716-c7524070e155" containerName="revoke-keys" Nov 02 22:20:53.076890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.076887 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ed30e3-6b87-4249-9716-c7524070e155" containerName="revoke-keys" Nov 02 22:20:53.077063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.077035 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ed30e3-6b87-4249-9716-c7524070e155" containerName="revoke-keys" Nov 02 22:20:53.079959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.079940 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.128999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.128968 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8263a4-openshift-default-dockercfg-t2p57\"" Nov 02 22:20:53.178261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.178228 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66"] Nov 02 22:20:53.214413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214381 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbkn\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-kube-api-access-ldbkn\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.214654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214430 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9cf88a-65ba-4646-bd38-179e399737db-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.214654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214449 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.214654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214503 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9cf88a-65ba-4646-bd38-179e399737db-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.214654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214631 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.214851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214829 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.214910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214895 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.214953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.214942 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.215036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.215021 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.315883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.315841 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbkn\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-kube-api-access-ldbkn\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.315897 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9cf88a-65ba-4646-bd38-179e399737db-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.315912 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.315927 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9cf88a-65ba-4646-bd38-179e399737db-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.315944 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316052 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316104 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316150 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316190 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316609 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316706 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.316777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316756 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.317060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.316785 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.317241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.317212 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9cf88a-65ba-4646-bd38-179e399737db-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.318712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.318682 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.318823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.318807 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9cf88a-65ba-4646-bd38-179e399737db-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.374249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.374165 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.374414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.374278 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbkn\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-kube-api-access-ldbkn\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.394290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.394253 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:53.607545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.607517 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:53.720774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.720684 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2fbc73-440a-4209-8eec-a256c1310417-kube-api-access-dxwll\" (UniqueName: \"kubernetes.io/projected/1d2fbc73-440a-4209-8eec-a256c1310417-kube-api-access-dxwll\") pod \"1d2fbc73-440a-4209-8eec-a256c1310417\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " Nov 02 22:20:53.720948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.720827 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-xks-ca\") pod \"1d2fbc73-440a-4209-8eec-a256c1310417\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " Nov 02 22:20:53.720948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.720879 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-openshift-ca\") pod \"1d2fbc73-440a-4209-8eec-a256c1310417\" (UID: \"1d2fbc73-440a-4209-8eec-a256c1310417\") " Nov 02 22:20:53.721599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.721539 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-openshift-ca" pod "1d2fbc73-440a-4209-8eec-a256c1310417" (UID: "1d2fbc73-440a-4209-8eec-a256c1310417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:53.723110 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.723088 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-xks-ca" pod "1d2fbc73-440a-4209-8eec-a256c1310417" (UID: "1d2fbc73-440a-4209-8eec-a256c1310417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:53.723340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.723317 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2fbc73-440a-4209-8eec-a256c1310417-kube-api-access-dxwll" pod "1d2fbc73-440a-4209-8eec-a256c1310417" (UID: "1d2fbc73-440a-4209-8eec-a256c1310417"). InnerVolumeSpecName "kube-api-access-dxwll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:53.821997 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.821965 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:53.821997 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.821995 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2fbc73-440a-4209-8eec-a256c1310417-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:53.821997 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.822005 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwll\" (UniqueName: \"kubernetes.io/projected/1d2fbc73-440a-4209-8eec-a256c1310417-kube-api-access-dxwll\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:53.869998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.869967 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66"] Nov 02 22:20:53.871070 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:20:53.871041 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9cf88a_65ba_4646_bd38_179e399737db.slice/crio-77f39e8afa4c4da89f2ca610102c08c8809069a4198cb5b6e80b8fd49625e13a WatchSource:0}: Error finding container 77f39e8afa4c4da89f2ca610102c08c8809069a4198cb5b6e80b8fd49625e13a: Status 404 returned error can't find the container with id 77f39e8afa4c4da89f2ca610102c08c8809069a4198cb5b6e80b8fd49625e13a Nov 02 22:20:53.873737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.873708 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:20:53.873812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.873772 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:20:53.873812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:53.873802 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:20:54.459493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:54.459464 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" Nov 02 22:20:54.459955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:54.459459 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f" event={"ID":"1d2fbc73-440a-4209-8eec-a256c1310417","Type":"ContainerDied","Data":"ef0db662193022e8e17fb8da0141e41948d0a80d156a349f402b7f3ebfae08dd"} Nov 02 22:20:54.459955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:54.459654 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef0db662193022e8e17fb8da0141e41948d0a80d156a349f402b7f3ebfae08dd" Nov 02 22:20:54.461295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:54.461271 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" event={"ID":"bb9cf88a-65ba-4646-bd38-179e399737db","Type":"ContainerStarted","Data":"065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a"} Nov 02 22:20:54.461485 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:54.461320 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" event={"ID":"bb9cf88a-65ba-4646-bd38-179e399737db","Type":"ContainerStarted","Data":"77f39e8afa4c4da89f2ca610102c08c8809069a4198cb5b6e80b8fd49625e13a"} Nov 02 22:20:54.575531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:54.575473 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podStartSLOduration=1.575455287 podStartE2EDuration="1.575455287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:54.574475234 +0000 UTC m=+890.839468465" watchObservedRunningTime="2026-11-02 22:20:54.575455287 +0000 UTC m=+890.840448507" Nov 02 22:20:55.394675 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:55.394630 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:20:55.396439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:55.396416 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 22:20:55.396489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:55.396468 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 22:20:56.395685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:56.395643 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 22:20:56.396076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:56.395707 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 22:20:56.490863 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:20:56.490826 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:20:57.201208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:57.201173 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c"] Nov 02 22:20:57.201423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:57.201396 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" containerID="cri-o://c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1" gracePeriod=30 Nov 02 22:20:57.402024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:57.401982 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 22:20:57.406112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:57.406064 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 22:20:58.395773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:58.395731 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 22:20:58.396040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:58.396017 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 22:20:59.395279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:59.395235 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 22:20:59.395785 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:20:59.395299 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 22:21:00.395033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:00.394990 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 22:21:00.395237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:00.395065 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 22:21:01.399693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:01.399660 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:21:01.400094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:01.399935 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:21:01.400678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:01.400660 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:21:02.461504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.461481 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:21:02.527267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.527230 5109 generic.go:356] "Generic (PLEG): container finished" podID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerID="c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1" exitCode=0 Nov 02 22:21:02.527458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.527285 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" Nov 02 22:21:02.527458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.527323 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" event={"ID":"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c","Type":"ContainerDied","Data":"c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1"} Nov 02 22:21:02.527458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.527391 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c" event={"ID":"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c","Type":"ContainerDied","Data":"f324670010580818cdcf353e96880a36a604af34d5c7def2a5952ae0014cacea"} Nov 02 22:21:02.527458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.527419 5109 scope.go:122] "RemoveContainer" containerID="c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1" Nov 02 22:21:02.537552 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.537530 5109 scope.go:122] "RemoveContainer" containerID="c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1" Nov 02 22:21:02.537873 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:21:02.537856 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1\": container with ID starting with c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1 not found: ID does not exist" containerID="c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1" Nov 02 22:21:02.537926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.537883 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1"} err="failed to get container status \"c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1\": rpc error: code = NotFound desc = could not find container \"c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1\": container with ID starting with c49fe21343cf2b44ec4e058018c0e64282ec5b8a377587095b79047ab4566aa1 not found: ID does not exist" Nov 02 22:21:02.557389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557356 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-credential-socket\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557411 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-kube-api-access-ks8cv\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-kube-api-access-ks8cv\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557461 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-socket\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557500 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-envoy\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557520 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-podinfo\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557545 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istiod-ca-cert\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557597 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-certs\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557719 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-credential-socket" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.557837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557799 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-token\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-token\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557854 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-data\") pod \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\" (UID: \"7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c\") " Nov 02 22:21:02.557983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557857 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-certs" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.557983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.557874 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-socket" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.558145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.558063 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istiod-ca-cert" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:02.558195 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.558166 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-data" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.558646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.558621 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.558744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.558651 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.558744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.558666 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.558744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.558680 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.558744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.558695 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.560039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.560016 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-kube-api-access-ks8cv" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "kube-api-access-ks8cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:02.560195 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.560170 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-envoy" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.560412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.560393 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-token" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:02.560412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.560403 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-podinfo" pod "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" (UID: "7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:21:02.659773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.659735 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.659773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.659774 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.660029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.659790 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.660029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.659803 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8cv\" (UniqueName: \"kubernetes.io/projected/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c-kube-api-access-ks8cv\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.944879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.944842 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c"] Nov 02 22:21:02.949599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:02.949513 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-hfn4c"] Nov 02 22:21:03.892153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:03.892121 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" path="/var/lib/kubelet/pods/7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c/volumes" Nov 02 22:21:07.566276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.566241 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln"] Nov 02 22:21:07.566809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.566795 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" Nov 02 22:21:07.566854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.566812 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" Nov 02 22:21:07.566854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.566830 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1d2fbc73-440a-4209-8eec-a256c1310417" containerName="revoke-keys" Nov 02 22:21:07.566854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.566835 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2fbc73-440a-4209-8eec-a256c1310417" containerName="revoke-keys" Nov 02 22:21:07.566952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.566933 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d00dd36-6f5d-42a9-b59d-1acdb1db6a9c" containerName="istio-proxy" Nov 02 22:21:07.566952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.566943 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2fbc73-440a-4209-8eec-a256c1310417" containerName="revoke-keys" Nov 02 22:21:07.571761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.571742 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.621582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.621538 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-49a557-openshift-default-dockercfg-lgv8v\"" Nov 02 22:21:07.674892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.674857 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln"] Nov 02 22:21:07.682929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.682902 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf"] Nov 02 22:21:07.687176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.687155 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.716472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716430 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.716666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716477 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4t2\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-kube-api-access-6w4t2\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.716666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716562 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.716666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716653 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.716775 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716729 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a45c2e-274a-4f64-acf1-7836436b6c50-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.716812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716791 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.716901 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716877 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.716954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.716918 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.717044 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.717029 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.737023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.736995 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:07.737208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.737192 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:21:07.778355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.778321 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf"] Nov 02 22:21:07.818276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818195 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818227 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818265 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818297 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.818554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818339 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.818554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818459 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818508 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4t2\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-kube-api-access-6w4t2\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818616 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818674 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818712 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a45c2e-274a-4f64-acf1-7836436b6c50-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818756 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtgx\" (UniqueName: \"kubernetes.io/projected/e3646f5b-eccd-4afe-8074-2889bb75a558-kube-api-access-8xtgx\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.818987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818830 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.818987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.818979 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.819113 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.819092 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.819259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.819231 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.819384 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.819257 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.819468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.819443 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a45c2e-274a-4f64-acf1-7836436b6c50-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.820800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.820777 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.821071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.821054 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.876526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.876494 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.876714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.876692 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4t2\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-kube-api-access-6w4t2\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.885527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.885501 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:07.919894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.919850 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.920069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.919908 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.920069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.920038 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtgx\" (UniqueName: \"kubernetes.io/projected/e3646f5b-eccd-4afe-8074-2889bb75a558-kube-api-access-8xtgx\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.920615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.920587 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.922621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.922595 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:07.978219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:07.978191 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtgx\" (UniqueName: \"kubernetes.io/projected/e3646f5b-eccd-4afe-8074-2889bb75a558-kube-api-access-8xtgx\") pod \"maas-api-revoke-sub-fcc6b28e-2nmsf\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:08.004121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.004083 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:08.308590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.308541 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln"] Nov 02 22:21:08.309723 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:08.309698 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a45c2e_274a_4f64_acf1_7836436b6c50.slice/crio-442bd0c6ea05106bbc994bcad2403cc1654d15fdb8e8a6deb696287ae3edc541 WatchSource:0}: Error finding container 442bd0c6ea05106bbc994bcad2403cc1654d15fdb8e8a6deb696287ae3edc541: Status 404 returned error can't find the container with id 442bd0c6ea05106bbc994bcad2403cc1654d15fdb8e8a6deb696287ae3edc541 Nov 02 22:21:08.312232 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.312198 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:08.312321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.312266 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:08.312321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.312304 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:08.399018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.398697 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf"] Nov 02 22:21:08.400508 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:08.400480 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3646f5b_eccd_4afe_8074_2889bb75a558.slice/crio-d5f2d5bb87460d724175fd81faaea30d45cf49ba7134d9e6f8a39d924cd6f8b8 WatchSource:0}: Error finding container d5f2d5bb87460d724175fd81faaea30d45cf49ba7134d9e6f8a39d924cd6f8b8: Status 404 returned error can't find the container with id d5f2d5bb87460d724175fd81faaea30d45cf49ba7134d9e6f8a39d924cd6f8b8 Nov 02 22:21:08.566028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.565936 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" event={"ID":"e3646f5b-eccd-4afe-8074-2889bb75a558","Type":"ContainerStarted","Data":"d5f2d5bb87460d724175fd81faaea30d45cf49ba7134d9e6f8a39d924cd6f8b8"} Nov 02 22:21:08.567883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.567853 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" event={"ID":"39a45c2e-274a-4f64-acf1-7836436b6c50","Type":"ContainerStarted","Data":"8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3"} Nov 02 22:21:08.568268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.567895 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" event={"ID":"39a45c2e-274a-4f64-acf1-7836436b6c50","Type":"ContainerStarted","Data":"442bd0c6ea05106bbc994bcad2403cc1654d15fdb8e8a6deb696287ae3edc541"} Nov 02 22:21:08.684611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.684540 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podStartSLOduration=1.6845253869999999 podStartE2EDuration="1.684525387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:08.681696614 +0000 UTC m=+904.946689935" watchObservedRunningTime="2026-11-02 22:21:08.684525387 +0000 UTC m=+904.949518607" Nov 02 22:21:08.886135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:08.886107 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:09.576828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:09.576774 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" event={"ID":"e3646f5b-eccd-4afe-8074-2889bb75a558","Type":"ContainerStarted","Data":"8e18f7ea67686b625037dc4ecd1c844028e6fa5f1ca8a0ff5dd732247dc06a38"} Nov 02 22:21:09.694685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:09.694627 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" podStartSLOduration=2.694613391 podStartE2EDuration="2.694613391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:09.693205527 +0000 UTC m=+905.958198746" watchObservedRunningTime="2026-11-02 22:21:09.694613391 +0000 UTC m=+905.959606609" Nov 02 22:21:09.888660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:09.888560 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:21:09.888660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:09.888633 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 22:21:10.584319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:10.584278 5109 generic.go:356] "Generic (PLEG): container finished" podID="e3646f5b-eccd-4afe-8074-2889bb75a558" containerID="8e18f7ea67686b625037dc4ecd1c844028e6fa5f1ca8a0ff5dd732247dc06a38" exitCode=0 Nov 02 22:21:10.584788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:10.584361 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" event={"ID":"e3646f5b-eccd-4afe-8074-2889bb75a558","Type":"ContainerDied","Data":"8e18f7ea67686b625037dc4ecd1c844028e6fa5f1ca8a0ff5dd732247dc06a38"} Nov 02 22:21:10.886474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:10.886389 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:21:10.886474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:10.886443 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 22:21:11.784035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.784003 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:11.886196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.886113 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:21:11.886196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.886166 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 22:21:11.890171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.890146 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3646f5b-eccd-4afe-8074-2889bb75a558-kube-api-access-8xtgx\" (UniqueName: \"kubernetes.io/projected/e3646f5b-eccd-4afe-8074-2889bb75a558-kube-api-access-8xtgx\") pod \"e3646f5b-eccd-4afe-8074-2889bb75a558\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " Nov 02 22:21:11.890289 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.890221 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-xks-ca\") pod \"e3646f5b-eccd-4afe-8074-2889bb75a558\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " Nov 02 22:21:11.890354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.890322 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-openshift-ca\") pod \"e3646f5b-eccd-4afe-8074-2889bb75a558\" (UID: \"e3646f5b-eccd-4afe-8074-2889bb75a558\") " Nov 02 22:21:11.890801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.890765 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-openshift-ca" pod "e3646f5b-eccd-4afe-8074-2889bb75a558" (UID: "e3646f5b-eccd-4afe-8074-2889bb75a558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:11.890994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.890961 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:11.892472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.892448 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3646f5b-eccd-4afe-8074-2889bb75a558-kube-api-access-8xtgx" pod "e3646f5b-eccd-4afe-8074-2889bb75a558" (UID: "e3646f5b-eccd-4afe-8074-2889bb75a558"). InnerVolumeSpecName "kube-api-access-8xtgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:11.892664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.892468 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-xks-ca" pod "e3646f5b-eccd-4afe-8074-2889bb75a558" (UID: "e3646f5b-eccd-4afe-8074-2889bb75a558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:11.992940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.992896 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xtgx\" (UniqueName: \"kubernetes.io/projected/e3646f5b-eccd-4afe-8074-2889bb75a558-kube-api-access-8xtgx\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:11.992940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:11.992945 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3646f5b-eccd-4afe-8074-2889bb75a558-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:12.604032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.604002 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" Nov 02 22:21:12.604303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.604042 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf" event={"ID":"e3646f5b-eccd-4afe-8074-2889bb75a558","Type":"ContainerDied","Data":"d5f2d5bb87460d724175fd81faaea30d45cf49ba7134d9e6f8a39d924cd6f8b8"} Nov 02 22:21:12.604411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.604328 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f2d5bb87460d724175fd81faaea30d45cf49ba7134d9e6f8a39d924cd6f8b8" Nov 02 22:21:12.749717 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.749643 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj"] Nov 02 22:21:12.750853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.750823 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e3646f5b-eccd-4afe-8074-2889bb75a558" containerName="revoke-keys" Nov 02 22:21:12.751028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.751013 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3646f5b-eccd-4afe-8074-2889bb75a558" containerName="revoke-keys" Nov 02 22:21:12.751276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.751263 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3646f5b-eccd-4afe-8074-2889bb75a558" containerName="revoke-keys" Nov 02 22:21:12.755178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.755155 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.794016 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.793981 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj"] Nov 02 22:21:12.802127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.802045 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.802127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.802101 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.802361 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.802172 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl26r\" (UniqueName: \"kubernetes.io/projected/10bcda0b-bc5d-4131-b15c-16339702817a-kube-api-access-bl26r\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.804490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.804422 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:12.804895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.804838 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:21:12.886060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.885979 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:21:12.886060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.886030 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 22:21:12.903597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.903532 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.903597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.903602 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.903820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.903641 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl26r\" (UniqueName: \"kubernetes.io/projected/10bcda0b-bc5d-4131-b15c-16339702817a-kube-api-access-bl26r\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.904263 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.904238 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.906085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.906065 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:12.962479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:12.962442 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl26r\" (UniqueName: \"kubernetes.io/projected/10bcda0b-bc5d-4131-b15c-16339702817a-kube-api-access-bl26r\") pod \"maas-api-revoke-sub-026aaab9-trrtj\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:13.070641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:13.070605 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:13.460123 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:13.460080 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj"] Nov 02 22:21:13.461449 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:13.461418 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bcda0b_bc5d_4131_b15c_16339702817a.slice/crio-5e9d90dc6ad9274334332158f1726437199c2a4d50e8fd7b378f17f77c75836b WatchSource:0}: Error finding container 5e9d90dc6ad9274334332158f1726437199c2a4d50e8fd7b378f17f77c75836b: Status 404 returned error can't find the container with id 5e9d90dc6ad9274334332158f1726437199c2a4d50e8fd7b378f17f77c75836b Nov 02 22:21:13.610803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:13.610764 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" event={"ID":"10bcda0b-bc5d-4131-b15c-16339702817a","Type":"ContainerStarted","Data":"5e9d90dc6ad9274334332158f1726437199c2a4d50e8fd7b378f17f77c75836b"} Nov 02 22:21:13.888769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:13.888736 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 22:21:13.889204 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:13.888785 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 22:21:14.618188 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:14.618146 5109 generic.go:356] "Generic (PLEG): container finished" podID="10bcda0b-bc5d-4131-b15c-16339702817a" containerID="be93fad1c53cdcda9ffde4e65f872249de2e1f3833e1b53647dea52bcfb6ae15" exitCode=0 Nov 02 22:21:14.618403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:14.618225 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" event={"ID":"10bcda0b-bc5d-4131-b15c-16339702817a","Type":"ContainerDied","Data":"be93fad1c53cdcda9ffde4e65f872249de2e1f3833e1b53647dea52bcfb6ae15"} Nov 02 22:21:15.050026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.049982 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:15.050987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.050949 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:15.052137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.052105 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:21:15.773328 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.773301 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:15.834295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.834259 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-xks-ca\") pod \"10bcda0b-bc5d-4131-b15c-16339702817a\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " Nov 02 22:21:15.834478 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.834355 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bcda0b-bc5d-4131-b15c-16339702817a-kube-api-access-bl26r\" (UniqueName: \"kubernetes.io/projected/10bcda0b-bc5d-4131-b15c-16339702817a-kube-api-access-bl26r\") pod \"10bcda0b-bc5d-4131-b15c-16339702817a\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " Nov 02 22:21:15.834478 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.834384 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-openshift-ca\") pod \"10bcda0b-bc5d-4131-b15c-16339702817a\" (UID: \"10bcda0b-bc5d-4131-b15c-16339702817a\") " Nov 02 22:21:15.834857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.834833 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-openshift-ca" pod "10bcda0b-bc5d-4131-b15c-16339702817a" (UID: "10bcda0b-bc5d-4131-b15c-16339702817a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:15.836649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.836626 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-xks-ca" pod "10bcda0b-bc5d-4131-b15c-16339702817a" (UID: "10bcda0b-bc5d-4131-b15c-16339702817a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:15.836749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.836700 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bcda0b-bc5d-4131-b15c-16339702817a-kube-api-access-bl26r" pod "10bcda0b-bc5d-4131-b15c-16339702817a" (UID: "10bcda0b-bc5d-4131-b15c-16339702817a"). InnerVolumeSpecName "kube-api-access-bl26r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:15.935829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.935795 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:15.935829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.935826 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl26r\" (UniqueName: \"kubernetes.io/projected/10bcda0b-bc5d-4131-b15c-16339702817a-kube-api-access-bl26r\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:15.935829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:15.935837 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bcda0b-bc5d-4131-b15c-16339702817a-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:16.631314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:16.631281 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" Nov 02 22:21:16.631749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:16.631309 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj" event={"ID":"10bcda0b-bc5d-4131-b15c-16339702817a","Type":"ContainerDied","Data":"5e9d90dc6ad9274334332158f1726437199c2a4d50e8fd7b378f17f77c75836b"} Nov 02 22:21:16.631749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:16.631358 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9d90dc6ad9274334332158f1726437199c2a4d50e8fd7b378f17f77c75836b" Nov 02 22:21:18.641548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.641514 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv"] Nov 02 22:21:18.642216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.642187 5109 cpu_manager.go:414] "containerMap: removing container" podUID="10bcda0b-bc5d-4131-b15c-16339702817a" containerName="revoke-keys" Nov 02 22:21:18.642216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.642215 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bcda0b-bc5d-4131-b15c-16339702817a" containerName="revoke-keys" Nov 02 22:21:18.642438 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.642343 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bcda0b-bc5d-4131-b15c-16339702817a" containerName="revoke-keys" Nov 02 22:21:18.645809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.645788 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.695207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.695180 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:18.742022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.741991 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:21:18.744919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.744896 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv"] Nov 02 22:21:18.766677 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.766641 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.766821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.766801 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8jk\" (UniqueName: \"kubernetes.io/projected/8914c825-df1b-498f-b342-3593a4d2f331-kube-api-access-hc8jk\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.766878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.766843 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.867925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.867890 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.868101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.867981 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8jk\" (UniqueName: \"kubernetes.io/projected/8914c825-df1b-498f-b342-3593a4d2f331-kube-api-access-hc8jk\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.868101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.868009 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.868673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.868652 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.870370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.870353 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.926334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.926260 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8jk\" (UniqueName: \"kubernetes.io/projected/8914c825-df1b-498f-b342-3593a4d2f331-kube-api-access-hc8jk\") pod \"maas-api-revoke-sub-2360b032-jjcwv\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:18.961786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:18.961755 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:19.335132 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:19.335096 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv"] Nov 02 22:21:19.665730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:19.664211 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" event={"ID":"8914c825-df1b-498f-b342-3593a4d2f331","Type":"ContainerStarted","Data":"366b6602254511e8ab3d04ce338cc63737f1fe0b0308aeb02ca83f69f2d7e2c8"} Nov 02 22:21:19.665730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:19.664288 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" event={"ID":"8914c825-df1b-498f-b342-3593a4d2f331","Type":"ContainerStarted","Data":"7e3b7fa5a462416b840eda4672cea942032e4d75aa12762d3e61cea7f5912054"} Nov 02 22:21:19.818508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:19.818451 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" podStartSLOduration=1.818436571 podStartE2EDuration="1.818436571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:19.815983198 +0000 UTC m=+916.080976419" watchObservedRunningTime="2026-11-02 22:21:19.818436571 +0000 UTC m=+916.083429789" Nov 02 22:21:20.013741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.013697 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx"] Nov 02 22:21:20.017599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.017559 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.084278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.084241 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx"] Nov 02 22:21:20.099613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.098332 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.099613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.098516 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.099613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.098633 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ts8f\" (UniqueName: \"kubernetes.io/projected/832747a8-354a-4109-9fc5-b54c2d5808dd-kube-api-access-9ts8f\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.223595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.212904 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ts8f\" (UniqueName: \"kubernetes.io/projected/832747a8-354a-4109-9fc5-b54c2d5808dd-kube-api-access-9ts8f\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.223595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.213072 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.223595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.213248 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.223595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.215308 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.231089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.228366 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.294437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.294352 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ts8f\" (UniqueName: \"kubernetes.io/projected/832747a8-354a-4109-9fc5-b54c2d5808dd-kube-api-access-9ts8f\") pod \"maas-api-revoke-sub-ea1484f6-gwbpx\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.337104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.337062 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:20.678678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.677773 5109 generic.go:356] "Generic (PLEG): container finished" podID="8914c825-df1b-498f-b342-3593a4d2f331" containerID="366b6602254511e8ab3d04ce338cc63737f1fe0b0308aeb02ca83f69f2d7e2c8" exitCode=0 Nov 02 22:21:20.678678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.677855 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" event={"ID":"8914c825-df1b-498f-b342-3593a4d2f331","Type":"ContainerDied","Data":"366b6602254511e8ab3d04ce338cc63737f1fe0b0308aeb02ca83f69f2d7e2c8"} Nov 02 22:21:20.735673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:20.735627 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx"] Nov 02 22:21:20.737757 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:20.737726 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832747a8_354a_4109_9fc5_b54c2d5808dd.slice/crio-025ca9b947f93d4bd14e553c1708904364342b57b6777ab60e4b0edfdb4f2418 WatchSource:0}: Error finding container 025ca9b947f93d4bd14e553c1708904364342b57b6777ab60e4b0edfdb4f2418: Status 404 returned error can't find the container with id 025ca9b947f93d4bd14e553c1708904364342b57b6777ab60e4b0edfdb4f2418 Nov 02 22:21:21.691768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:21.691719 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" event={"ID":"832747a8-354a-4109-9fc5-b54c2d5808dd","Type":"ContainerStarted","Data":"e18234908bae927d828e2d9850801031a89b4366e5a05ce12838da949781f4be"} Nov 02 22:21:21.692263 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:21.691787 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" event={"ID":"832747a8-354a-4109-9fc5-b54c2d5808dd","Type":"ContainerStarted","Data":"025ca9b947f93d4bd14e553c1708904364342b57b6777ab60e4b0edfdb4f2418"} Nov 02 22:21:21.803067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:21.803002 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" podStartSLOduration=2.802984784 podStartE2EDuration="2.802984784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:21.802834485 +0000 UTC m=+918.067827703" watchObservedRunningTime="2026-11-02 22:21:21.802984784 +0000 UTC m=+918.067978008" Nov 02 22:21:21.959590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:21.952467 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:22.054471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.054434 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-openshift-ca\") pod \"8914c825-df1b-498f-b342-3593a4d2f331\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " Nov 02 22:21:22.054721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.054645 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-xks-ca\") pod \"8914c825-df1b-498f-b342-3593a4d2f331\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " Nov 02 22:21:22.054721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.054712 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8914c825-df1b-498f-b342-3593a4d2f331-kube-api-access-hc8jk\" (UniqueName: \"kubernetes.io/projected/8914c825-df1b-498f-b342-3593a4d2f331-kube-api-access-hc8jk\") pod \"8914c825-df1b-498f-b342-3593a4d2f331\" (UID: \"8914c825-df1b-498f-b342-3593a4d2f331\") " Nov 02 22:21:22.055660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.055612 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-openshift-ca" pod "8914c825-df1b-498f-b342-3593a4d2f331" (UID: "8914c825-df1b-498f-b342-3593a4d2f331"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:22.060090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.060049 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8914c825-df1b-498f-b342-3593a4d2f331-kube-api-access-hc8jk" pod "8914c825-df1b-498f-b342-3593a4d2f331" (UID: "8914c825-df1b-498f-b342-3593a4d2f331"). InnerVolumeSpecName "kube-api-access-hc8jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:22.061076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.060604 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-xks-ca" pod "8914c825-df1b-498f-b342-3593a4d2f331" (UID: "8914c825-df1b-498f-b342-3593a4d2f331"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:22.156300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.156260 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:22.156621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.156598 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8jk\" (UniqueName: \"kubernetes.io/projected/8914c825-df1b-498f-b342-3593a4d2f331-kube-api-access-hc8jk\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:22.156746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.156732 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8914c825-df1b-498f-b342-3593a4d2f331-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:22.658594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.658168 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk"] Nov 02 22:21:22.662619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.658905 5109 cpu_manager.go:414] "containerMap: removing container" podUID="8914c825-df1b-498f-b342-3593a4d2f331" containerName="revoke-keys" Nov 02 22:21:22.662619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.658928 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8914c825-df1b-498f-b342-3593a4d2f331" containerName="revoke-keys" Nov 02 22:21:22.662619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.659102 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="8914c825-df1b-498f-b342-3593a4d2f331" containerName="revoke-keys" Nov 02 22:21:22.662918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.662896 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.699919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.699893 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" Nov 02 22:21:22.700374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.699890 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv" event={"ID":"8914c825-df1b-498f-b342-3593a4d2f331","Type":"ContainerDied","Data":"7e3b7fa5a462416b840eda4672cea942032e4d75aa12762d3e61cea7f5912054"} Nov 02 22:21:22.700374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.700015 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e3b7fa5a462416b840eda4672cea942032e4d75aa12762d3e61cea7f5912054" Nov 02 22:21:22.701741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.701700 5109 generic.go:356] "Generic (PLEG): container finished" podID="832747a8-354a-4109-9fc5-b54c2d5808dd" containerID="e18234908bae927d828e2d9850801031a89b4366e5a05ce12838da949781f4be" exitCode=0 Nov 02 22:21:22.701868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.701772 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" event={"ID":"832747a8-354a-4109-9fc5-b54c2d5808dd","Type":"ContainerDied","Data":"e18234908bae927d828e2d9850801031a89b4366e5a05ce12838da949781f4be"} Nov 02 22:21:22.718542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.718509 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk"] Nov 02 22:21:22.763644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.763602 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tx92\" (UniqueName: \"kubernetes.io/projected/e45438ae-ff43-4c35-83d2-4f02237d6aac-kube-api-access-4tx92\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.763839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.763783 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.763931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.763865 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.865540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.865500 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.865768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.865619 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tx92\" (UniqueName: \"kubernetes.io/projected/e45438ae-ff43-4c35-83d2-4f02237d6aac-kube-api-access-4tx92\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.865836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.865783 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.866615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.866589 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.868235 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.868211 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.926802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.926719 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tx92\" (UniqueName: \"kubernetes.io/projected/e45438ae-ff43-4c35-83d2-4f02237d6aac-kube-api-access-4tx92\") pod \"maas-api-revoke-sub-7b5ceffc-jhspk\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:22.978756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:22.978724 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:23.352910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.352878 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk"] Nov 02 22:21:23.353970 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:23.353944 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45438ae_ff43_4c35_83d2_4f02237d6aac.slice/crio-28fe63eb26ed312be45ac2f0ec7c41da807eadcca7333df47e9b8eaf6b49cf7d WatchSource:0}: Error finding container 28fe63eb26ed312be45ac2f0ec7c41da807eadcca7333df47e9b8eaf6b49cf7d: Status 404 returned error can't find the container with id 28fe63eb26ed312be45ac2f0ec7c41da807eadcca7333df47e9b8eaf6b49cf7d Nov 02 22:21:23.709622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.709561 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" event={"ID":"e45438ae-ff43-4c35-83d2-4f02237d6aac","Type":"ContainerStarted","Data":"28445922a2717ccc95fe5a3ae30b8ffc2f667a9773f3d76142c06f1a30a35855"} Nov 02 22:21:23.710079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.709643 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" event={"ID":"e45438ae-ff43-4c35-83d2-4f02237d6aac","Type":"ContainerStarted","Data":"28fe63eb26ed312be45ac2f0ec7c41da807eadcca7333df47e9b8eaf6b49cf7d"} Nov 02 22:21:23.819798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.819752 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" podStartSLOduration=1.819738623 podStartE2EDuration="1.819738623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:23.819116257 +0000 UTC m=+920.084109476" watchObservedRunningTime="2026-11-02 22:21:23.819738623 +0000 UTC m=+920.084731842" Nov 02 22:21:23.849395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.849363 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:23.979713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.979676 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-xks-ca\") pod \"832747a8-354a-4109-9fc5-b54c2d5808dd\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " Nov 02 22:21:23.979936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.979748 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832747a8-354a-4109-9fc5-b54c2d5808dd-kube-api-access-9ts8f\" (UniqueName: \"kubernetes.io/projected/832747a8-354a-4109-9fc5-b54c2d5808dd-kube-api-access-9ts8f\") pod \"832747a8-354a-4109-9fc5-b54c2d5808dd\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " Nov 02 22:21:23.979936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.979798 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-openshift-ca\") pod \"832747a8-354a-4109-9fc5-b54c2d5808dd\" (UID: \"832747a8-354a-4109-9fc5-b54c2d5808dd\") " Nov 02 22:21:23.980253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.980223 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-openshift-ca" pod "832747a8-354a-4109-9fc5-b54c2d5808dd" (UID: "832747a8-354a-4109-9fc5-b54c2d5808dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:23.980443 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.980427 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:23.982080 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.982047 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-xks-ca" pod "832747a8-354a-4109-9fc5-b54c2d5808dd" (UID: "832747a8-354a-4109-9fc5-b54c2d5808dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:23.982863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:23.982838 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832747a8-354a-4109-9fc5-b54c2d5808dd-kube-api-access-9ts8f" pod "832747a8-354a-4109-9fc5-b54c2d5808dd" (UID: "832747a8-354a-4109-9fc5-b54c2d5808dd"). InnerVolumeSpecName "kube-api-access-9ts8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:24.081981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:24.081939 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832747a8-354a-4109-9fc5-b54c2d5808dd-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:24.081981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:24.081972 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ts8f\" (UniqueName: \"kubernetes.io/projected/832747a8-354a-4109-9fc5-b54c2d5808dd-kube-api-access-9ts8f\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:24.718078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:24.718044 5109 generic.go:356] "Generic (PLEG): container finished" podID="e45438ae-ff43-4c35-83d2-4f02237d6aac" containerID="28445922a2717ccc95fe5a3ae30b8ffc2f667a9773f3d76142c06f1a30a35855" exitCode=0 Nov 02 22:21:24.718554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:24.718146 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" event={"ID":"e45438ae-ff43-4c35-83d2-4f02237d6aac","Type":"ContainerDied","Data":"28445922a2717ccc95fe5a3ae30b8ffc2f667a9773f3d76142c06f1a30a35855"} Nov 02 22:21:24.719538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:24.719510 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" Nov 02 22:21:24.719676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:24.719547 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx" event={"ID":"832747a8-354a-4109-9fc5-b54c2d5808dd","Type":"ContainerDied","Data":"025ca9b947f93d4bd14e553c1708904364342b57b6777ab60e4b0edfdb4f2418"} Nov 02 22:21:24.719676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:24.719618 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="025ca9b947f93d4bd14e553c1708904364342b57b6777ab60e4b0edfdb4f2418" Nov 02 22:21:25.520172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.520132 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc"] Nov 02 22:21:25.520774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.520755 5109 cpu_manager.go:414] "containerMap: removing container" podUID="832747a8-354a-4109-9fc5-b54c2d5808dd" containerName="revoke-keys" Nov 02 22:21:25.520862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.520776 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832747a8-354a-4109-9fc5-b54c2d5808dd" containerName="revoke-keys" Nov 02 22:21:25.520895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.520872 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="832747a8-354a-4109-9fc5-b54c2d5808dd" containerName="revoke-keys" Nov 02 22:21:25.524429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.524408 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.580640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.580603 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc"] Nov 02 22:21:25.599320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.599285 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.599504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.599347 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.599586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.599528 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92swp\" (UniqueName: \"kubernetes.io/projected/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-kube-api-access-92swp\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.700937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.700903 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.701159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.700966 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.701159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.701077 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92swp\" (UniqueName: \"kubernetes.io/projected/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-kube-api-access-92swp\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.701691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.701663 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.703553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.703530 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.763213 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.763175 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92swp\" (UniqueName: \"kubernetes.io/projected/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-kube-api-access-92swp\") pod \"maas-api-revoke-sub-df893d0b-s9qzc\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.838269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.838238 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:25.882643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:25.882618 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:26.004621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.004584 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-xks-ca\") pod \"e45438ae-ff43-4c35-83d2-4f02237d6aac\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " Nov 02 22:21:26.004843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.004682 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45438ae-ff43-4c35-83d2-4f02237d6aac-kube-api-access-4tx92\" (UniqueName: \"kubernetes.io/projected/e45438ae-ff43-4c35-83d2-4f02237d6aac-kube-api-access-4tx92\") pod \"e45438ae-ff43-4c35-83d2-4f02237d6aac\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " Nov 02 22:21:26.004843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.004763 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-openshift-ca\") pod \"e45438ae-ff43-4c35-83d2-4f02237d6aac\" (UID: \"e45438ae-ff43-4c35-83d2-4f02237d6aac\") " Nov 02 22:21:26.005217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.005182 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-openshift-ca" pod "e45438ae-ff43-4c35-83d2-4f02237d6aac" (UID: "e45438ae-ff43-4c35-83d2-4f02237d6aac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:26.006904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.006874 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-xks-ca" pod "e45438ae-ff43-4c35-83d2-4f02237d6aac" (UID: "e45438ae-ff43-4c35-83d2-4f02237d6aac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:26.007468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.007447 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45438ae-ff43-4c35-83d2-4f02237d6aac-kube-api-access-4tx92" pod "e45438ae-ff43-4c35-83d2-4f02237d6aac" (UID: "e45438ae-ff43-4c35-83d2-4f02237d6aac"). InnerVolumeSpecName "kube-api-access-4tx92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:26.107013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.106933 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:26.107013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.106963 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tx92\" (UniqueName: \"kubernetes.io/projected/e45438ae-ff43-4c35-83d2-4f02237d6aac-kube-api-access-4tx92\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:26.107013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.106973 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45438ae-ff43-4c35-83d2-4f02237d6aac-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:26.223094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.223065 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc"] Nov 02 22:21:26.223952 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:26.223899 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc9a5a3_6d45_4675_a695_6304f3cc7dac.slice/crio-a277e5bb28737d491ecf5826d57f40279365bf773bc569fd5934c50e1c0d92c9 WatchSource:0}: Error finding container a277e5bb28737d491ecf5826d57f40279365bf773bc569fd5934c50e1c0d92c9: Status 404 returned error can't find the container with id a277e5bb28737d491ecf5826d57f40279365bf773bc569fd5934c50e1c0d92c9 Nov 02 22:21:26.491234 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:21:26.491196 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:21:26.733494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.733406 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" Nov 02 22:21:26.733494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.733438 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk" event={"ID":"e45438ae-ff43-4c35-83d2-4f02237d6aac","Type":"ContainerDied","Data":"28fe63eb26ed312be45ac2f0ec7c41da807eadcca7333df47e9b8eaf6b49cf7d"} Nov 02 22:21:26.733494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.733489 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28fe63eb26ed312be45ac2f0ec7c41da807eadcca7333df47e9b8eaf6b49cf7d" Nov 02 22:21:26.735199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.735171 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" event={"ID":"2cc9a5a3-6d45-4675-a695-6304f3cc7dac","Type":"ContainerStarted","Data":"ec35ec4046b8db999b1514ae0602e0868f7221d3db0be5e839068d561d9c18fb"} Nov 02 22:21:26.735329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.735227 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" event={"ID":"2cc9a5a3-6d45-4675-a695-6304f3cc7dac","Type":"ContainerStarted","Data":"a277e5bb28737d491ecf5826d57f40279365bf773bc569fd5934c50e1c0d92c9"} Nov 02 22:21:26.846725 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:26.846665 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" podStartSLOduration=1.846646422 podStartE2EDuration="1.846646422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:26.844901985 +0000 UTC m=+923.109895205" watchObservedRunningTime="2026-11-02 22:21:26.846646422 +0000 UTC m=+923.111639673" Nov 02 22:21:27.743522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:27.743489 5109 generic.go:356] "Generic (PLEG): container finished" podID="2cc9a5a3-6d45-4675-a695-6304f3cc7dac" containerID="ec35ec4046b8db999b1514ae0602e0868f7221d3db0be5e839068d561d9c18fb" exitCode=0 Nov 02 22:21:27.743758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:27.743602 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" event={"ID":"2cc9a5a3-6d45-4675-a695-6304f3cc7dac","Type":"ContainerDied","Data":"ec35ec4046b8db999b1514ae0602e0868f7221d3db0be5e839068d561d9c18fb"} Nov 02 22:21:28.926963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:28.926940 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:29.041111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.041078 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-xks-ca\") pod \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " Nov 02 22:21:29.041264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.041239 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-openshift-ca\") pod \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " Nov 02 22:21:29.041359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.041344 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-kube-api-access-92swp\" (UniqueName: \"kubernetes.io/projected/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-kube-api-access-92swp\") pod \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\" (UID: \"2cc9a5a3-6d45-4675-a695-6304f3cc7dac\") " Nov 02 22:21:29.041676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.041651 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-openshift-ca" pod "2cc9a5a3-6d45-4675-a695-6304f3cc7dac" (UID: "2cc9a5a3-6d45-4675-a695-6304f3cc7dac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:29.041960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.041905 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:29.043524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.043498 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-xks-ca" pod "2cc9a5a3-6d45-4675-a695-6304f3cc7dac" (UID: "2cc9a5a3-6d45-4675-a695-6304f3cc7dac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:29.043652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.043583 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-kube-api-access-92swp" pod "2cc9a5a3-6d45-4675-a695-6304f3cc7dac" (UID: "2cc9a5a3-6d45-4675-a695-6304f3cc7dac"). InnerVolumeSpecName "kube-api-access-92swp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:29.149857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.143965 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:29.149857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.144007 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92swp\" (UniqueName: \"kubernetes.io/projected/2cc9a5a3-6d45-4675-a695-6304f3cc7dac-kube-api-access-92swp\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:29.758052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.758020 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" Nov 02 22:21:29.758052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.758039 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc" event={"ID":"2cc9a5a3-6d45-4675-a695-6304f3cc7dac","Type":"ContainerDied","Data":"a277e5bb28737d491ecf5826d57f40279365bf773bc569fd5934c50e1c0d92c9"} Nov 02 22:21:29.758266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:29.758113 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a277e5bb28737d491ecf5826d57f40279365bf773bc569fd5934c50e1c0d92c9" Nov 02 22:21:30.429079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.429047 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl"] Nov 02 22:21:30.429671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.429644 5109 cpu_manager.go:414] "containerMap: removing container" podUID="2cc9a5a3-6d45-4675-a695-6304f3cc7dac" containerName="revoke-keys" Nov 02 22:21:30.429671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.429663 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc9a5a3-6d45-4675-a695-6304f3cc7dac" containerName="revoke-keys" Nov 02 22:21:30.429757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.429715 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e45438ae-ff43-4c35-83d2-4f02237d6aac" containerName="revoke-keys" Nov 02 22:21:30.429757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.429720 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45438ae-ff43-4c35-83d2-4f02237d6aac" containerName="revoke-keys" Nov 02 22:21:30.429855 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.429844 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc9a5a3-6d45-4675-a695-6304f3cc7dac" containerName="revoke-keys" Nov 02 22:21:30.429895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.429867 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45438ae-ff43-4c35-83d2-4f02237d6aac" containerName="revoke-keys" Nov 02 22:21:30.433546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.433527 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.530781 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.530741 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:30.530963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.530858 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:21:30.533937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.533909 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl"] Nov 02 22:21:30.561774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.561736 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.561934 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.561806 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7snn\" (UniqueName: \"kubernetes.io/projected/20a15ec9-ab64-46b6-abd5-996db792d644-kube-api-access-q7snn\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.561934 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.561902 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.663282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.663239 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7snn\" (UniqueName: \"kubernetes.io/projected/20a15ec9-ab64-46b6-abd5-996db792d644-kube-api-access-q7snn\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.663472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.663295 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.663472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.663382 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.664022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.664001 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.665926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.665902 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.721718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.721635 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7snn\" (UniqueName: \"kubernetes.io/projected/20a15ec9-ab64-46b6-abd5-996db792d644-kube-api-access-q7snn\") pod \"maas-api-revoke-sub-e6fee2d8-2m9jl\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:30.749403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:30.749364 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:31.121641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:31.121607 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl"] Nov 02 22:21:31.122212 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:31.122188 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a15ec9_ab64_46b6_abd5_996db792d644.slice/crio-7780900ee0f821ee647f2085498ae1486aec7d7b721ed082457faa4093f3cfe6 WatchSource:0}: Error finding container 7780900ee0f821ee647f2085498ae1486aec7d7b721ed082457faa4093f3cfe6: Status 404 returned error can't find the container with id 7780900ee0f821ee647f2085498ae1486aec7d7b721ed082457faa4093f3cfe6 Nov 02 22:21:31.774117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:31.774076 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" event={"ID":"20a15ec9-ab64-46b6-abd5-996db792d644","Type":"ContainerStarted","Data":"aff18e9d9582585d4bf72848d17d4d1410a2ea6612c5cf9ff60b4a940549b733"} Nov 02 22:21:31.774652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:31.774150 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" event={"ID":"20a15ec9-ab64-46b6-abd5-996db792d644","Type":"ContainerStarted","Data":"7780900ee0f821ee647f2085498ae1486aec7d7b721ed082457faa4093f3cfe6"} Nov 02 22:21:31.883590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:31.883517 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" podStartSLOduration=1.883501187 podStartE2EDuration="1.883501187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:31.883052679 +0000 UTC m=+928.148045898" watchObservedRunningTime="2026-11-02 22:21:31.883501187 +0000 UTC m=+928.148494405" Nov 02 22:21:32.782170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:32.782135 5109 generic.go:356] "Generic (PLEG): container finished" podID="20a15ec9-ab64-46b6-abd5-996db792d644" containerID="aff18e9d9582585d4bf72848d17d4d1410a2ea6612c5cf9ff60b4a940549b733" exitCode=0 Nov 02 22:21:32.782623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:32.782227 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" event={"ID":"20a15ec9-ab64-46b6-abd5-996db792d644","Type":"ContainerDied","Data":"aff18e9d9582585d4bf72848d17d4d1410a2ea6612c5cf9ff60b4a940549b733"} Nov 02 22:21:33.483482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.483442 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf"] Nov 02 22:21:33.487752 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.487727 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.544943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.544901 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf"] Nov 02 22:21:33.599717 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.599672 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.599922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.599861 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8b4\" (UniqueName: \"kubernetes.io/projected/1c9d9621-5e45-40f6-bc56-d514fae48378-kube-api-access-nv8b4\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.600018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.600000 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.701110 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.701059 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.701305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.701187 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.701305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.701273 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8b4\" (UniqueName: \"kubernetes.io/projected/1c9d9621-5e45-40f6-bc56-d514fae48378-kube-api-access-nv8b4\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.702043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.701999 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.704312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.704290 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.759715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.759684 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8b4\" (UniqueName: \"kubernetes.io/projected/1c9d9621-5e45-40f6-bc56-d514fae48378-kube-api-access-nv8b4\") pod \"maas-api-revoke-sub-be817925-4cltf\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.801777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.801745 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:33.958615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:33.958585 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:34.106137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.106045 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a15ec9-ab64-46b6-abd5-996db792d644-kube-api-access-q7snn\" (UniqueName: \"kubernetes.io/projected/20a15ec9-ab64-46b6-abd5-996db792d644-kube-api-access-q7snn\") pod \"20a15ec9-ab64-46b6-abd5-996db792d644\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " Nov 02 22:21:34.106137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.106125 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-xks-ca\") pod \"20a15ec9-ab64-46b6-abd5-996db792d644\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " Nov 02 22:21:34.106372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.106201 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-openshift-ca\") pod \"20a15ec9-ab64-46b6-abd5-996db792d644\" (UID: \"20a15ec9-ab64-46b6-abd5-996db792d644\") " Nov 02 22:21:34.106713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.106677 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-openshift-ca" pod "20a15ec9-ab64-46b6-abd5-996db792d644" (UID: "20a15ec9-ab64-46b6-abd5-996db792d644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:34.108721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.108692 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a15ec9-ab64-46b6-abd5-996db792d644-kube-api-access-q7snn" pod "20a15ec9-ab64-46b6-abd5-996db792d644" (UID: "20a15ec9-ab64-46b6-abd5-996db792d644"). InnerVolumeSpecName "kube-api-access-q7snn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:34.108874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.108854 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-xks-ca" pod "20a15ec9-ab64-46b6-abd5-996db792d644" (UID: "20a15ec9-ab64-46b6-abd5-996db792d644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:34.200876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.200824 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf"] Nov 02 22:21:34.202022 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:34.201992 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9d9621_5e45_40f6_bc56_d514fae48378.slice/crio-63d2b1d69bfcad3be341b04f38ff763158d3eef74ffc9d2a7dd8b1945635cfb4 WatchSource:0}: Error finding container 63d2b1d69bfcad3be341b04f38ff763158d3eef74ffc9d2a7dd8b1945635cfb4: Status 404 returned error can't find the container with id 63d2b1d69bfcad3be341b04f38ff763158d3eef74ffc9d2a7dd8b1945635cfb4 Nov 02 22:21:34.207744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.207719 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.207816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.207751 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7snn\" (UniqueName: \"kubernetes.io/projected/20a15ec9-ab64-46b6-abd5-996db792d644-kube-api-access-q7snn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.207816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.207767 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a15ec9-ab64-46b6-abd5-996db792d644-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.652135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.652097 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc"] Nov 02 22:21:34.652700 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.652686 5109 cpu_manager.go:414] "containerMap: removing container" podUID="20a15ec9-ab64-46b6-abd5-996db792d644" containerName="revoke-keys" Nov 02 22:21:34.652769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.652702 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a15ec9-ab64-46b6-abd5-996db792d644" containerName="revoke-keys" Nov 02 22:21:34.652831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.652820 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a15ec9-ab64-46b6-abd5-996db792d644" containerName="revoke-keys" Nov 02 22:21:34.656369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.656347 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.711821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.711724 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc"] Nov 02 22:21:34.796031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.795997 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" event={"ID":"1c9d9621-5e45-40f6-bc56-d514fae48378","Type":"ContainerStarted","Data":"844f7db90ca062f927670a109681b6466c4c16afa5dce02a8484341338ddf74a"} Nov 02 22:21:34.796208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.796054 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" event={"ID":"1c9d9621-5e45-40f6-bc56-d514fae48378","Type":"ContainerStarted","Data":"63d2b1d69bfcad3be341b04f38ff763158d3eef74ffc9d2a7dd8b1945635cfb4"} Nov 02 22:21:34.797427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.797394 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" Nov 02 22:21:34.797558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.797455 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl" event={"ID":"20a15ec9-ab64-46b6-abd5-996db792d644","Type":"ContainerDied","Data":"7780900ee0f821ee647f2085498ae1486aec7d7b721ed082457faa4093f3cfe6"} Nov 02 22:21:34.797558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.797508 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7780900ee0f821ee647f2085498ae1486aec7d7b721ed082457faa4093f3cfe6" Nov 02 22:21:34.813909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.813885 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.814304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.814040 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmpw\" (UniqueName: \"kubernetes.io/projected/be01902d-224f-4351-9be1-dc86d62b774a-kube-api-access-cgmpw\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.814304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.814106 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.905765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.905704 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" podStartSLOduration=1.9056875739999999 podStartE2EDuration="1.905687574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:34.904166132 +0000 UTC m=+931.169159354" watchObservedRunningTime="2026-11-02 22:21:34.905687574 +0000 UTC m=+931.170680800" Nov 02 22:21:34.915950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.915884 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.916160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.916060 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmpw\" (UniqueName: \"kubernetes.io/projected/be01902d-224f-4351-9be1-dc86d62b774a-kube-api-access-cgmpw\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.916160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.916105 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.916840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.916819 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.918830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.918808 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:34.980520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:34.980438 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmpw\" (UniqueName: \"kubernetes.io/projected/be01902d-224f-4351-9be1-dc86d62b774a-kube-api-access-cgmpw\") pod \"maas-api-revoke-sub-77b63bd6-tfrsc\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:35.270439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:35.270403 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:35.644926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:35.644898 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc"] Nov 02 22:21:35.647301 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:35.647274 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe01902d_224f_4351_9be1_dc86d62b774a.slice/crio-9f4885f50162d78c45270dd66988097508292820b96e66651f6d48217896845e WatchSource:0}: Error finding container 9f4885f50162d78c45270dd66988097508292820b96e66651f6d48217896845e: Status 404 returned error can't find the container with id 9f4885f50162d78c45270dd66988097508292820b96e66651f6d48217896845e Nov 02 22:21:35.805640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:35.805606 5109 generic.go:356] "Generic (PLEG): container finished" podID="1c9d9621-5e45-40f6-bc56-d514fae48378" containerID="844f7db90ca062f927670a109681b6466c4c16afa5dce02a8484341338ddf74a" exitCode=0 Nov 02 22:21:35.805800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:35.805703 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" event={"ID":"1c9d9621-5e45-40f6-bc56-d514fae48378","Type":"ContainerDied","Data":"844f7db90ca062f927670a109681b6466c4c16afa5dce02a8484341338ddf74a"} Nov 02 22:21:35.807061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:35.807036 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" event={"ID":"be01902d-224f-4351-9be1-dc86d62b774a","Type":"ContainerStarted","Data":"9f4885f50162d78c45270dd66988097508292820b96e66651f6d48217896845e"} Nov 02 22:21:36.445543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.445503 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk"] Nov 02 22:21:36.450052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.450027 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.506502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.506467 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk"] Nov 02 22:21:36.555168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.555135 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.555168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.555178 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn569\" (UniqueName: \"kubernetes.io/projected/89374f18-0e9b-4fbf-a615-a1f5526c66a6-kube-api-access-cn569\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.555405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.555382 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.656487 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.656448 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.656692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.656495 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn569\" (UniqueName: \"kubernetes.io/projected/89374f18-0e9b-4fbf-a615-a1f5526c66a6-kube-api-access-cn569\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.656741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.656718 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.657144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.657123 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.659256 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.659229 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.714748 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.714717 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn569\" (UniqueName: \"kubernetes.io/projected/89374f18-0e9b-4fbf-a615-a1f5526c66a6-kube-api-access-cn569\") pod \"maas-api-revoke-sub-97ef7566-9sqvk\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.763843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.763802 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:36.816023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.815981 5109 generic.go:356] "Generic (PLEG): container finished" podID="be01902d-224f-4351-9be1-dc86d62b774a" containerID="4f5142513e1494d976746a844b1c8f67aa1a356a4dc32222275496cf68486484" exitCode=0 Nov 02 22:21:36.816201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.816091 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" event={"ID":"be01902d-224f-4351-9be1-dc86d62b774a","Type":"ContainerDied","Data":"4f5142513e1494d976746a844b1c8f67aa1a356a4dc32222275496cf68486484"} Nov 02 22:21:36.997170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:36.997143 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:37.060442 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.060404 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-openshift-ca\") pod \"1c9d9621-5e45-40f6-bc56-d514fae48378\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " Nov 02 22:21:37.060640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.060461 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-xks-ca\") pod \"1c9d9621-5e45-40f6-bc56-d514fae48378\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " Nov 02 22:21:37.060640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.060498 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9d9621-5e45-40f6-bc56-d514fae48378-kube-api-access-nv8b4\" (UniqueName: \"kubernetes.io/projected/1c9d9621-5e45-40f6-bc56-d514fae48378-kube-api-access-nv8b4\") pod \"1c9d9621-5e45-40f6-bc56-d514fae48378\" (UID: \"1c9d9621-5e45-40f6-bc56-d514fae48378\") " Nov 02 22:21:37.060904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.060880 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-openshift-ca" pod "1c9d9621-5e45-40f6-bc56-d514fae48378" (UID: "1c9d9621-5e45-40f6-bc56-d514fae48378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:37.062947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.062919 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9d9621-5e45-40f6-bc56-d514fae48378-kube-api-access-nv8b4" pod "1c9d9621-5e45-40f6-bc56-d514fae48378" (UID: "1c9d9621-5e45-40f6-bc56-d514fae48378"). InnerVolumeSpecName "kube-api-access-nv8b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:37.063170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.063143 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-xks-ca" pod "1c9d9621-5e45-40f6-bc56-d514fae48378" (UID: "1c9d9621-5e45-40f6-bc56-d514fae48378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:37.136668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.136628 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk"] Nov 02 22:21:37.136853 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:37.136823 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89374f18_0e9b_4fbf_a615_a1f5526c66a6.slice/crio-4d0719eec11e821403aef543c848ea0856118454f34d72ca0de606e25f810ee3 WatchSource:0}: Error finding container 4d0719eec11e821403aef543c848ea0856118454f34d72ca0de606e25f810ee3: Status 404 returned error can't find the container with id 4d0719eec11e821403aef543c848ea0856118454f34d72ca0de606e25f810ee3 Nov 02 22:21:37.161635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.161607 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:37.161635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.161635 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9d9621-5e45-40f6-bc56-d514fae48378-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:37.161787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.161644 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv8b4\" (UniqueName: \"kubernetes.io/projected/1c9d9621-5e45-40f6-bc56-d514fae48378-kube-api-access-nv8b4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:37.824712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.824682 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" Nov 02 22:21:37.825135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.824680 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf" event={"ID":"1c9d9621-5e45-40f6-bc56-d514fae48378","Type":"ContainerDied","Data":"63d2b1d69bfcad3be341b04f38ff763158d3eef74ffc9d2a7dd8b1945635cfb4"} Nov 02 22:21:37.825135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.824845 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d2b1d69bfcad3be341b04f38ff763158d3eef74ffc9d2a7dd8b1945635cfb4" Nov 02 22:21:37.827400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.826864 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" event={"ID":"89374f18-0e9b-4fbf-a615-a1f5526c66a6","Type":"ContainerStarted","Data":"46f93a88d6017ec1e09fbfd4877e433cba6009fcf4b891b9f5e8ff53031b4682"} Nov 02 22:21:37.827400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.826925 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" event={"ID":"89374f18-0e9b-4fbf-a615-a1f5526c66a6","Type":"ContainerStarted","Data":"4d0719eec11e821403aef543c848ea0856118454f34d72ca0de606e25f810ee3"} Nov 02 22:21:37.939232 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.939177 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" podStartSLOduration=1.939160233 podStartE2EDuration="1.939160233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:37.935515359 +0000 UTC m=+934.200508569" watchObservedRunningTime="2026-11-02 22:21:37.939160233 +0000 UTC m=+934.204153463" Nov 02 22:21:37.979109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:37.979038 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:38.070807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.070776 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-openshift-ca\") pod \"be01902d-224f-4351-9be1-dc86d62b774a\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " Nov 02 22:21:38.070972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.070900 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be01902d-224f-4351-9be1-dc86d62b774a-kube-api-access-cgmpw\" (UniqueName: \"kubernetes.io/projected/be01902d-224f-4351-9be1-dc86d62b774a-kube-api-access-cgmpw\") pod \"be01902d-224f-4351-9be1-dc86d62b774a\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " Nov 02 22:21:38.070972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.070942 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-xks-ca\") pod \"be01902d-224f-4351-9be1-dc86d62b774a\" (UID: \"be01902d-224f-4351-9be1-dc86d62b774a\") " Nov 02 22:21:38.071277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.071248 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-openshift-ca" pod "be01902d-224f-4351-9be1-dc86d62b774a" (UID: "be01902d-224f-4351-9be1-dc86d62b774a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:38.073262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.073234 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-xks-ca" pod "be01902d-224f-4351-9be1-dc86d62b774a" (UID: "be01902d-224f-4351-9be1-dc86d62b774a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:38.073375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.073267 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be01902d-224f-4351-9be1-dc86d62b774a-kube-api-access-cgmpw" pod "be01902d-224f-4351-9be1-dc86d62b774a" (UID: "be01902d-224f-4351-9be1-dc86d62b774a"). InnerVolumeSpecName "kube-api-access-cgmpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:38.172308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.172273 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgmpw\" (UniqueName: \"kubernetes.io/projected/be01902d-224f-4351-9be1-dc86d62b774a-kube-api-access-cgmpw\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:38.172308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.172305 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:38.172308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.172316 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be01902d-224f-4351-9be1-dc86d62b774a-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:38.836377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.836337 5109 generic.go:356] "Generic (PLEG): container finished" podID="89374f18-0e9b-4fbf-a615-a1f5526c66a6" containerID="46f93a88d6017ec1e09fbfd4877e433cba6009fcf4b891b9f5e8ff53031b4682" exitCode=0 Nov 02 22:21:38.837705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.837433 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" event={"ID":"89374f18-0e9b-4fbf-a615-a1f5526c66a6","Type":"ContainerDied","Data":"46f93a88d6017ec1e09fbfd4877e433cba6009fcf4b891b9f5e8ff53031b4682"} Nov 02 22:21:38.839904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.839876 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" Nov 02 22:21:38.839904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.839884 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc" event={"ID":"be01902d-224f-4351-9be1-dc86d62b774a","Type":"ContainerDied","Data":"9f4885f50162d78c45270dd66988097508292820b96e66651f6d48217896845e"} Nov 02 22:21:38.840101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:38.839954 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4885f50162d78c45270dd66988097508292820b96e66651f6d48217896845e" Nov 02 22:21:40.002089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.002061 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:40.095467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.095433 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89374f18-0e9b-4fbf-a615-a1f5526c66a6-kube-api-access-cn569\" (UniqueName: \"kubernetes.io/projected/89374f18-0e9b-4fbf-a615-a1f5526c66a6-kube-api-access-cn569\") pod \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " Nov 02 22:21:40.095716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.095613 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-xks-ca\") pod \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " Nov 02 22:21:40.095792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.095733 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-openshift-ca\") pod \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\" (UID: \"89374f18-0e9b-4fbf-a615-a1f5526c66a6\") " Nov 02 22:21:40.096081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.096056 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-openshift-ca" pod "89374f18-0e9b-4fbf-a615-a1f5526c66a6" (UID: "89374f18-0e9b-4fbf-a615-a1f5526c66a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:40.097766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.097717 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-xks-ca" pod "89374f18-0e9b-4fbf-a615-a1f5526c66a6" (UID: "89374f18-0e9b-4fbf-a615-a1f5526c66a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:40.097833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.097796 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89374f18-0e9b-4fbf-a615-a1f5526c66a6-kube-api-access-cn569" pod "89374f18-0e9b-4fbf-a615-a1f5526c66a6" (UID: "89374f18-0e9b-4fbf-a615-a1f5526c66a6"). InnerVolumeSpecName "kube-api-access-cn569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:40.196762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.196716 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:40.196762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.196751 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89374f18-0e9b-4fbf-a615-a1f5526c66a6-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:40.196762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.196761 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn569\" (UniqueName: \"kubernetes.io/projected/89374f18-0e9b-4fbf-a615-a1f5526c66a6-kube-api-access-cn569\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:40.854990 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.854954 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" Nov 02 22:21:40.854990 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.854984 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk" event={"ID":"89374f18-0e9b-4fbf-a615-a1f5526c66a6","Type":"ContainerDied","Data":"4d0719eec11e821403aef543c848ea0856118454f34d72ca0de606e25f810ee3"} Nov 02 22:21:40.855249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:40.855031 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d0719eec11e821403aef543c848ea0856118454f34d72ca0de606e25f810ee3" Nov 02 22:21:42.589589 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.589529 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp"] Nov 02 22:21:42.593592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.590382 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1c9d9621-5e45-40f6-bc56-d514fae48378" containerName="revoke-keys" Nov 02 22:21:42.593592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.590405 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9d9621-5e45-40f6-bc56-d514fae48378" containerName="revoke-keys" Nov 02 22:21:42.593592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.590423 5109 cpu_manager.go:414] "containerMap: removing container" podUID="be01902d-224f-4351-9be1-dc86d62b774a" containerName="revoke-keys" Nov 02 22:21:42.593592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.590431 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be01902d-224f-4351-9be1-dc86d62b774a" containerName="revoke-keys" Nov 02 22:21:42.593592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.590472 5109 cpu_manager.go:414] "containerMap: removing container" podUID="89374f18-0e9b-4fbf-a615-a1f5526c66a6" containerName="revoke-keys" Nov 02 22:21:42.593592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.590479 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89374f18-0e9b-4fbf-a615-a1f5526c66a6" containerName="revoke-keys" Nov 02 22:21:42.610593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.606946 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="89374f18-0e9b-4fbf-a615-a1f5526c66a6" containerName="revoke-keys" Nov 02 22:21:42.610593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.606988 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="be01902d-224f-4351-9be1-dc86d62b774a" containerName="revoke-keys" Nov 02 22:21:42.610593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.607000 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9d9621-5e45-40f6-bc56-d514fae48378" containerName="revoke-keys" Nov 02 22:21:42.617315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.617282 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.643294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.643256 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm"] Nov 02 22:21:42.738837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.738796 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.744454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.741824 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.744734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.744711 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.744886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.744871 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c722bd3-a137-4489-9330-037fd83d4d5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.745048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.745032 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v94t\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-kube-api-access-8v94t\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.745160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.745147 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.745253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.745242 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.745389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.745377 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.745515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.745503 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.745659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.745644 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.824428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.824391 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp"] Nov 02 22:21:42.832727 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.832700 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm"] Nov 02 22:21:42.857391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.855714 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.857685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.857661 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.858177 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.858149 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pm5q\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-kube-api-access-5pm5q\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.932995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.856155 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.932995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.932934 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933014 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933082 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.933257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933142 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933172 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933239 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.933522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933317 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.933522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933374 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933458 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.933522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933505 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.933742 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933531 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933742 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933591 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c722bd3-a137-4489-9330-037fd83d4d5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.933742 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933620 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933742 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933714 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:42.933943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.933785 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v94t\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-kube-api-access-8v94t\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.948163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.934748 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.965645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.952302 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c722bd3-a137-4489-9330-037fd83d4d5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.965645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.952619 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.965645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.958006 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.965645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.962424 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.989097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.980911 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.993157 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.993124 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:42.996591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:42.994956 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v94t\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-kube-api-access-8v94t\") pod \"maas-default-gateway-openshift-default-76f4876667-rclwp\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035007 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035070 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035160 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035266 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035310 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035380 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035459 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pm5q\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-kube-api-access-5pm5q\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035496 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.035590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.035540 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.039834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.036407 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.044010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.041948 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.044010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.042238 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.044010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.042590 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.044010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.043945 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.045170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.045123 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.051988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.051938 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.093810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.093754 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.100268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.100231 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pm5q\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-kube-api-access-5pm5q\") pod \"maas-default-gateway-openshift-default-76f4876667-d6shm\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.132955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.132868 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:43.244867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.244833 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:43.599625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.596787 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm"] Nov 02 22:21:43.603546 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:43.603505 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6248b1_95a4_4446_8abf_55b4cc516ea7.slice/crio-9d115b551d0fcddc77f41ab22ee1350fc843a2f5a7d5c30f7ca20d94157b4616 WatchSource:0}: Error finding container 9d115b551d0fcddc77f41ab22ee1350fc843a2f5a7d5c30f7ca20d94157b4616: Status 404 returned error can't find the container with id 9d115b551d0fcddc77f41ab22ee1350fc843a2f5a7d5c30f7ca20d94157b4616 Nov 02 22:21:43.607212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.607167 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:43.607291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.607269 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:43.607339 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.607322 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:43.724609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.723636 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp"] Nov 02 22:21:43.763715 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:43.754225 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c722bd3_a137_4489_9330_037fd83d4d5b.slice/crio-7393103bdc0171523b2f081b5e5ab07fcc2271cf3d330549cccfcf18a197784c WatchSource:0}: Error finding container 7393103bdc0171523b2f081b5e5ab07fcc2271cf3d330549cccfcf18a197784c: Status 404 returned error can't find the container with id 7393103bdc0171523b2f081b5e5ab07fcc2271cf3d330549cccfcf18a197784c Nov 02 22:21:43.763715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.762101 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:43.763715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.762220 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:43.763715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:43.762265 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:21:44.014626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.013705 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" event={"ID":"3c722bd3-a137-4489-9330-037fd83d4d5b","Type":"ContainerStarted","Data":"29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0"} Nov 02 22:21:44.014626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.013779 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" event={"ID":"3c722bd3-a137-4489-9330-037fd83d4d5b","Type":"ContainerStarted","Data":"7393103bdc0171523b2f081b5e5ab07fcc2271cf3d330549cccfcf18a197784c"} Nov 02 22:21:44.019598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.019547 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" event={"ID":"7c6248b1-95a4-4446-8abf-55b4cc516ea7","Type":"ContainerStarted","Data":"1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc"} Nov 02 22:21:44.019774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.019627 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" event={"ID":"7c6248b1-95a4-4446-8abf-55b4cc516ea7","Type":"ContainerStarted","Data":"9d115b551d0fcddc77f41ab22ee1350fc843a2f5a7d5c30f7ca20d94157b4616"} Nov 02 22:21:44.130800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.130662 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podStartSLOduration=2.130641838 podStartE2EDuration="2.130641838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:44.128359574 +0000 UTC m=+940.393352786" watchObservedRunningTime="2026-11-02 22:21:44.130641838 +0000 UTC m=+940.395635074" Nov 02 22:21:44.137406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.137374 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:44.138188 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.138161 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.138402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.138376 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:44.246599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.246486 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podStartSLOduration=2.246466435 podStartE2EDuration="2.246466435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:44.244557867 +0000 UTC m=+940.509551087" watchObservedRunningTime="2026-11-02 22:21:44.246466435 +0000 UTC m=+940.511459653" Nov 02 22:21:44.253861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.253100 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:44.253861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.253282 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.253861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:44.253325 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:45.137599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:45.134108 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:45.137599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:45.134172 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:45.264753 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:45.264711 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:45.268709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:45.268661 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:46.145032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:46.144992 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:46.145686 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:46.145648 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:46.251596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:46.250204 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:46.251596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:46.250277 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:47.159597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.147024 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:47.159597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.147103 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:47.261106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.261065 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:47.261348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.261320 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:47.283833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.283785 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29"] Nov 02 22:21:47.289650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.289623 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.304695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.300909 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29"] Nov 02 22:21:47.372608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.364897 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:47.372608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.365138 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:21:47.378658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.375736 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.378658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.375871 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqtkg\" (UniqueName: \"kubernetes.io/projected/dc3017f9-4e3e-46b3-9000-826e54b789f5-kube-api-access-dqtkg\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.378658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.375908 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.478144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.478036 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.478330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.478142 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqtkg\" (UniqueName: \"kubernetes.io/projected/dc3017f9-4e3e-46b3-9000-826e54b789f5-kube-api-access-dqtkg\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.478330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.478181 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.479341 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.479310 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.481127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.481106 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.549594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.549321 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqtkg\" (UniqueName: \"kubernetes.io/projected/dc3017f9-4e3e-46b3-9000-826e54b789f5-kube-api-access-dqtkg\") pod \"maas-api-revoke-sub-028b1a3f-g4f29\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:47.610597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:47.610392 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:48.102166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:48.102138 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29"] Nov 02 22:21:48.102751 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:48.102715 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3017f9_4e3e_46b3_9000_826e54b789f5.slice/crio-517e35ed91d39052411d4147fedf6e4e1fd9a424491e77ce4823594efc35da92 WatchSource:0}: Error finding container 517e35ed91d39052411d4147fedf6e4e1fd9a424491e77ce4823594efc35da92: Status 404 returned error can't find the container with id 517e35ed91d39052411d4147fedf6e4e1fd9a424491e77ce4823594efc35da92 Nov 02 22:21:48.138363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:48.136973 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:48.138363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:48.137033 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:48.249593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:48.248417 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:48.249593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:48.248477 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:49.099456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.099420 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" event={"ID":"dc3017f9-4e3e-46b3-9000-826e54b789f5","Type":"ContainerStarted","Data":"c1a20afc81f4908586bb1c7bf2dff07c5e657436d2803d3eb3aecf6da40fb6bd"} Nov 02 22:21:49.099663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.099472 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" event={"ID":"dc3017f9-4e3e-46b3-9000-826e54b789f5","Type":"ContainerStarted","Data":"517e35ed91d39052411d4147fedf6e4e1fd9a424491e77ce4823594efc35da92"} Nov 02 22:21:49.136079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.136037 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:49.136381 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.136351 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:49.220524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.220471 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" podStartSLOduration=2.220455877 podStartE2EDuration="2.220455877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:49.21994363 +0000 UTC m=+945.484936851" watchObservedRunningTime="2026-11-02 22:21:49.220455877 +0000 UTC m=+945.485449096" Nov 02 22:21:49.250067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.250030 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:49.250667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.250636 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:49.967597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.966602 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h"] Nov 02 22:21:49.984350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.984316 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:49.992924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:49.992891 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h"] Nov 02 22:21:50.108920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.108876 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.108920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.108915 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.109237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.109218 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbw2\" (UniqueName: \"kubernetes.io/projected/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-kube-api-access-6wbw2\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.114711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.114677 5109 generic.go:356] "Generic (PLEG): container finished" podID="dc3017f9-4e3e-46b3-9000-826e54b789f5" containerID="c1a20afc81f4908586bb1c7bf2dff07c5e657436d2803d3eb3aecf6da40fb6bd" exitCode=0 Nov 02 22:21:50.114840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.114759 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" event={"ID":"dc3017f9-4e3e-46b3-9000-826e54b789f5","Type":"ContainerDied","Data":"c1a20afc81f4908586bb1c7bf2dff07c5e657436d2803d3eb3aecf6da40fb6bd"} Nov 02 22:21:50.137595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.135118 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:50.137595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.135195 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:50.210275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.210237 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbw2\" (UniqueName: \"kubernetes.io/projected/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-kube-api-access-6wbw2\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.210475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.210304 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.210475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.210330 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.211138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.211112 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.212933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.212914 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.246212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.246111 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:50.246212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.246172 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:50.268776 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.268733 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbw2\" (UniqueName: \"kubernetes.io/projected/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-kube-api-access-6wbw2\") pod \"maas-api-revoke-sub-ff92eb48-gh68h\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.303168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.303141 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:50.690973 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:50.690923 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4efbcd_b337_4d2f_bf44_e993c14b23a0.slice/crio-4a9e739c7d537564cd5bd43e79a7c536e885fa28fab500a213e12be4634e68f7 WatchSource:0}: Error finding container 4a9e739c7d537564cd5bd43e79a7c536e885fa28fab500a213e12be4634e68f7: Status 404 returned error can't find the container with id 4a9e739c7d537564cd5bd43e79a7c536e885fa28fab500a213e12be4634e68f7 Nov 02 22:21:50.691163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:50.691054 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h"] Nov 02 22:21:51.125485 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.125353 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" event={"ID":"ef4efbcd-b337-4d2f-bf44-e993c14b23a0","Type":"ContainerStarted","Data":"e8a244d89e0236df5e8e6053c658ce5ffdaf453d9c02e67e30e46a0c74541107"} Nov 02 22:21:51.125485 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.125421 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" event={"ID":"ef4efbcd-b337-4d2f-bf44-e993c14b23a0","Type":"ContainerStarted","Data":"4a9e739c7d537564cd5bd43e79a7c536e885fa28fab500a213e12be4634e68f7"} Nov 02 22:21:51.133795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.133763 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d6shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:21:51.133925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.133818 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 22:21:51.235266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.235044 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" podStartSLOduration=2.235026595 podStartE2EDuration="2.235026595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:51.234539433 +0000 UTC m=+947.499532654" watchObservedRunningTime="2026-11-02 22:21:51.235026595 +0000 UTC m=+947.500019815" Nov 02 22:21:51.246346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.246295 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rclwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 22:21:51.246596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.246348 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 22:21:51.312433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.312400 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:51.321636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.321603 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-xks-ca\") pod \"dc3017f9-4e3e-46b3-9000-826e54b789f5\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " Nov 02 22:21:51.323962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.323931 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-xks-ca" pod "dc3017f9-4e3e-46b3-9000-826e54b789f5" (UID: "dc3017f9-4e3e-46b3-9000-826e54b789f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:51.423089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.423054 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3017f9-4e3e-46b3-9000-826e54b789f5-kube-api-access-dqtkg\" (UniqueName: \"kubernetes.io/projected/dc3017f9-4e3e-46b3-9000-826e54b789f5-kube-api-access-dqtkg\") pod \"dc3017f9-4e3e-46b3-9000-826e54b789f5\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " Nov 02 22:21:51.423286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.423133 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-openshift-ca\") pod \"dc3017f9-4e3e-46b3-9000-826e54b789f5\" (UID: \"dc3017f9-4e3e-46b3-9000-826e54b789f5\") " Nov 02 22:21:51.423626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.423557 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-openshift-ca" pod "dc3017f9-4e3e-46b3-9000-826e54b789f5" (UID: "dc3017f9-4e3e-46b3-9000-826e54b789f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:51.423626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.423613 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:51.427474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.427300 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3017f9-4e3e-46b3-9000-826e54b789f5-kube-api-access-dqtkg" pod "dc3017f9-4e3e-46b3-9000-826e54b789f5" (UID: "dc3017f9-4e3e-46b3-9000-826e54b789f5"). InnerVolumeSpecName "kube-api-access-dqtkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:51.524969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.524929 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqtkg\" (UniqueName: \"kubernetes.io/projected/dc3017f9-4e3e-46b3-9000-826e54b789f5-kube-api-access-dqtkg\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:51.524969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:51.524964 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3017f9-4e3e-46b3-9000-826e54b789f5-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:52.139646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.138907 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" Nov 02 22:21:52.139646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.138917 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29" event={"ID":"dc3017f9-4e3e-46b3-9000-826e54b789f5","Type":"ContainerDied","Data":"517e35ed91d39052411d4147fedf6e4e1fd9a424491e77ce4823594efc35da92"} Nov 02 22:21:52.139646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.139018 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517e35ed91d39052411d4147fedf6e4e1fd9a424491e77ce4823594efc35da92" Nov 02 22:21:52.143441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.143365 5109 generic.go:356] "Generic (PLEG): container finished" podID="ef4efbcd-b337-4d2f-bf44-e993c14b23a0" containerID="e8a244d89e0236df5e8e6053c658ce5ffdaf453d9c02e67e30e46a0c74541107" exitCode=0 Nov 02 22:21:52.143441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.143392 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" event={"ID":"ef4efbcd-b337-4d2f-bf44-e993c14b23a0","Type":"ContainerDied","Data":"e8a244d89e0236df5e8e6053c658ce5ffdaf453d9c02e67e30e46a0c74541107"} Nov 02 22:21:52.649691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.649435 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:52.650212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.650131 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:52.651043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.651019 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:21:52.705626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.705555 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:52.706058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.706030 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:52.707440 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:52.707420 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:21:53.315398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.315372 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:53.349592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.349525 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-kube-api-access-6wbw2\" (UniqueName: \"kubernetes.io/projected/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-kube-api-access-6wbw2\") pod \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " Nov 02 22:21:53.349771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.349675 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-xks-ca\") pod \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " Nov 02 22:21:53.349771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.349714 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-openshift-ca\") pod \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\" (UID: \"ef4efbcd-b337-4d2f-bf44-e993c14b23a0\") " Nov 02 22:21:53.350164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.350126 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-openshift-ca" pod "ef4efbcd-b337-4d2f-bf44-e993c14b23a0" (UID: "ef4efbcd-b337-4d2f-bf44-e993c14b23a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:53.351840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.351814 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-kube-api-access-6wbw2" pod "ef4efbcd-b337-4d2f-bf44-e993c14b23a0" (UID: "ef4efbcd-b337-4d2f-bf44-e993c14b23a0"). InnerVolumeSpecName "kube-api-access-6wbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:53.351951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.351895 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-xks-ca" pod "ef4efbcd-b337-4d2f-bf44-e993c14b23a0" (UID: "ef4efbcd-b337-4d2f-bf44-e993c14b23a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:53.452114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.452018 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:53.452114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.452063 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:53.452114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:53.452080 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wbw2\" (UniqueName: \"kubernetes.io/projected/ef4efbcd-b337-4d2f-bf44-e993c14b23a0-kube-api-access-6wbw2\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:54.159718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:54.159631 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" Nov 02 22:21:54.159718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:54.159650 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h" event={"ID":"ef4efbcd-b337-4d2f-bf44-e993c14b23a0","Type":"ContainerDied","Data":"4a9e739c7d537564cd5bd43e79a7c536e885fa28fab500a213e12be4634e68f7"} Nov 02 22:21:54.159718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:54.159707 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a9e739c7d537564cd5bd43e79a7c536e885fa28fab500a213e12be4634e68f7" Nov 02 22:21:55.326161 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.326126 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf"] Nov 02 22:21:55.326778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.326740 5109 cpu_manager.go:414] "containerMap: removing container" podUID="dc3017f9-4e3e-46b3-9000-826e54b789f5" containerName="revoke-keys" Nov 02 22:21:55.326778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.326756 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3017f9-4e3e-46b3-9000-826e54b789f5" containerName="revoke-keys" Nov 02 22:21:55.326778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.326774 5109 cpu_manager.go:414] "containerMap: removing container" podUID="ef4efbcd-b337-4d2f-bf44-e993c14b23a0" containerName="revoke-keys" Nov 02 22:21:55.326778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.326779 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4efbcd-b337-4d2f-bf44-e993c14b23a0" containerName="revoke-keys" Nov 02 22:21:55.326949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.326867 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3017f9-4e3e-46b3-9000-826e54b789f5" containerName="revoke-keys" Nov 02 22:21:55.326949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.326878 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4efbcd-b337-4d2f-bf44-e993c14b23a0" containerName="revoke-keys" Nov 02 22:21:55.331749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.331718 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.374155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.374108 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.374359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.374287 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.374359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.374320 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svss\" (UniqueName: \"kubernetes.io/projected/4881034b-843a-4e1f-a37d-ba4c37fed87c-kube-api-access-7svss\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.387003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.386972 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf"] Nov 02 22:21:55.428054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.428016 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:55.428240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.428070 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:21:55.475414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.475380 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.475598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.475497 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.475598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.475532 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svss\" (UniqueName: \"kubernetes.io/projected/4881034b-843a-4e1f-a37d-ba4c37fed87c-kube-api-access-7svss\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.476221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.476201 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.478007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.477985 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.533774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.533743 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svss\" (UniqueName: \"kubernetes.io/projected/4881034b-843a-4e1f-a37d-ba4c37fed87c-kube-api-access-7svss\") pod \"maas-api-revoke-sub-87aa09ae-msgkf\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:55.647885 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:55.647795 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:56.019392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:56.019356 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf"] Nov 02 22:21:56.020554 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:21:56.020522 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4881034b_843a_4e1f_a37d_ba4c37fed87c.slice/crio-f2802e2619132d2653e47b4a3f7aac625bdf0e02f458591bd7355eda6825ee4e WatchSource:0}: Error finding container f2802e2619132d2653e47b4a3f7aac625bdf0e02f458591bd7355eda6825ee4e: Status 404 returned error can't find the container with id f2802e2619132d2653e47b4a3f7aac625bdf0e02f458591bd7355eda6825ee4e Nov 02 22:21:56.174350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:56.174308 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" event={"ID":"4881034b-843a-4e1f-a37d-ba4c37fed87c","Type":"ContainerStarted","Data":"f2802e2619132d2653e47b4a3f7aac625bdf0e02f458591bd7355eda6825ee4e"} Nov 02 22:21:56.490964 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:21:56.490928 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:21:57.183343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:57.183241 5109 generic.go:356] "Generic (PLEG): container finished" podID="4881034b-843a-4e1f-a37d-ba4c37fed87c" containerID="7d76973a734b2355c4411cecc9d7917afa32df1bb5cd2dcf7741c9c7a81392f8" exitCode=0 Nov 02 22:21:57.183343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:57.183306 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" event={"ID":"4881034b-843a-4e1f-a37d-ba4c37fed87c","Type":"ContainerDied","Data":"7d76973a734b2355c4411cecc9d7917afa32df1bb5cd2dcf7741c9c7a81392f8"} Nov 02 22:21:58.347982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.347956 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:58.408282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.408249 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-openshift-ca\") pod \"4881034b-843a-4e1f-a37d-ba4c37fed87c\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " Nov 02 22:21:58.408442 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.408363 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-xks-ca\") pod \"4881034b-843a-4e1f-a37d-ba4c37fed87c\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " Nov 02 22:21:58.408614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.408555 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4881034b-843a-4e1f-a37d-ba4c37fed87c-kube-api-access-7svss\" (UniqueName: \"kubernetes.io/projected/4881034b-843a-4e1f-a37d-ba4c37fed87c-kube-api-access-7svss\") pod \"4881034b-843a-4e1f-a37d-ba4c37fed87c\" (UID: \"4881034b-843a-4e1f-a37d-ba4c37fed87c\") " Nov 02 22:21:58.408780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.408631 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-openshift-ca" pod "4881034b-843a-4e1f-a37d-ba4c37fed87c" (UID: "4881034b-843a-4e1f-a37d-ba4c37fed87c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:58.409018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.409000 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:58.410708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.410685 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-xks-ca" pod "4881034b-843a-4e1f-a37d-ba4c37fed87c" (UID: "4881034b-843a-4e1f-a37d-ba4c37fed87c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:58.410795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.410767 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4881034b-843a-4e1f-a37d-ba4c37fed87c-kube-api-access-7svss" pod "4881034b-843a-4e1f-a37d-ba4c37fed87c" (UID: "4881034b-843a-4e1f-a37d-ba4c37fed87c"). InnerVolumeSpecName "kube-api-access-7svss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:58.510323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.510284 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4881034b-843a-4e1f-a37d-ba4c37fed87c-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:58.510323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:58.510323 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svss\" (UniqueName: \"kubernetes.io/projected/4881034b-843a-4e1f-a37d-ba4c37fed87c-kube-api-access-7svss\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:59.200988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:59.200950 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" Nov 02 22:21:59.201166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:59.200958 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf" event={"ID":"4881034b-843a-4e1f-a37d-ba4c37fed87c","Type":"ContainerDied","Data":"f2802e2619132d2653e47b4a3f7aac625bdf0e02f458591bd7355eda6825ee4e"} Nov 02 22:21:59.201166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:21:59.201089 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2802e2619132d2653e47b4a3f7aac625bdf0e02f458591bd7355eda6825ee4e" Nov 02 22:22:05.282737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.282705 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk"] Nov 02 22:22:05.283454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.283423 5109 cpu_manager.go:414] "containerMap: removing container" podUID="4881034b-843a-4e1f-a37d-ba4c37fed87c" containerName="revoke-keys" Nov 02 22:22:05.283454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.283447 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4881034b-843a-4e1f-a37d-ba4c37fed87c" containerName="revoke-keys" Nov 02 22:22:05.283664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.283648 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="4881034b-843a-4e1f-a37d-ba4c37fed87c" containerName="revoke-keys" Nov 02 22:22:05.288222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.288202 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.338284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.338254 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:22:05.384746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.384713 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:22:05.388326 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.388299 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk"] Nov 02 22:22:05.438707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.438670 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.438707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.438708 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vt5r\" (UniqueName: \"kubernetes.io/projected/6953deba-db7d-44a8-97e9-f356fff60ff4-kube-api-access-6vt5r\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.438952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.438847 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.540368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.540263 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.540368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.540321 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vt5r\" (UniqueName: \"kubernetes.io/projected/6953deba-db7d-44a8-97e9-f356fff60ff4-kube-api-access-6vt5r\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.540621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.540396 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.541085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.541055 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.542881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.542861 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.600432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.600397 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vt5r\" (UniqueName: \"kubernetes.io/projected/6953deba-db7d-44a8-97e9-f356fff60ff4-kube-api-access-6vt5r\") pod \"maas-api-revoke-sub-664d0c6d-gzjgk\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.601921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.601906 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:05.976534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:05.976501 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk"] Nov 02 22:22:05.977078 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:22:05.977052 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6953deba_db7d_44a8_97e9_f356fff60ff4.slice/crio-55f647891e244d779ddac7c09201f8ceb9775c04ce5072e3a6f878311e3855b8 WatchSource:0}: Error finding container 55f647891e244d779ddac7c09201f8ceb9775c04ce5072e3a6f878311e3855b8: Status 404 returned error can't find the container with id 55f647891e244d779ddac7c09201f8ceb9775c04ce5072e3a6f878311e3855b8 Nov 02 22:22:06.267103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:06.267058 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" event={"ID":"6953deba-db7d-44a8-97e9-f356fff60ff4","Type":"ContainerStarted","Data":"6d96984a08941be52b99e555190f05c0888be1792c67ab61b6862b384488eea1"} Nov 02 22:22:06.267253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:06.267126 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" event={"ID":"6953deba-db7d-44a8-97e9-f356fff60ff4","Type":"ContainerStarted","Data":"55f647891e244d779ddac7c09201f8ceb9775c04ce5072e3a6f878311e3855b8"} Nov 02 22:22:06.378358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:06.378293 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" podStartSLOduration=1.378278952 podStartE2EDuration="1.378278952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:06.376634798 +0000 UTC m=+962.641628014" watchObservedRunningTime="2026-11-02 22:22:06.378278952 +0000 UTC m=+962.643272171" Nov 02 22:22:07.279635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:07.279600 5109 generic.go:356] "Generic (PLEG): container finished" podID="6953deba-db7d-44a8-97e9-f356fff60ff4" containerID="6d96984a08941be52b99e555190f05c0888be1792c67ab61b6862b384488eea1" exitCode=0 Nov 02 22:22:07.279815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:07.279741 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" event={"ID":"6953deba-db7d-44a8-97e9-f356fff60ff4","Type":"ContainerDied","Data":"6d96984a08941be52b99e555190f05c0888be1792c67ab61b6862b384488eea1"} Nov 02 22:22:08.447561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.447528 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:08.573793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.573694 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-xks-ca\") pod \"6953deba-db7d-44a8-97e9-f356fff60ff4\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " Nov 02 22:22:08.573793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.573760 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6953deba-db7d-44a8-97e9-f356fff60ff4-kube-api-access-6vt5r\" (UniqueName: \"kubernetes.io/projected/6953deba-db7d-44a8-97e9-f356fff60ff4-kube-api-access-6vt5r\") pod \"6953deba-db7d-44a8-97e9-f356fff60ff4\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " Nov 02 22:22:08.573793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.573788 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-openshift-ca\") pod \"6953deba-db7d-44a8-97e9-f356fff60ff4\" (UID: \"6953deba-db7d-44a8-97e9-f356fff60ff4\") " Nov 02 22:22:08.574249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.574221 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-openshift-ca" pod "6953deba-db7d-44a8-97e9-f356fff60ff4" (UID: "6953deba-db7d-44a8-97e9-f356fff60ff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:08.576142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.576113 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-xks-ca" pod "6953deba-db7d-44a8-97e9-f356fff60ff4" (UID: "6953deba-db7d-44a8-97e9-f356fff60ff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:08.576629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.576604 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6953deba-db7d-44a8-97e9-f356fff60ff4-kube-api-access-6vt5r" pod "6953deba-db7d-44a8-97e9-f356fff60ff4" (UID: "6953deba-db7d-44a8-97e9-f356fff60ff4"). InnerVolumeSpecName "kube-api-access-6vt5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:08.675085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.675050 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:08.675085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.675086 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vt5r\" (UniqueName: \"kubernetes.io/projected/6953deba-db7d-44a8-97e9-f356fff60ff4-kube-api-access-6vt5r\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:08.675315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:08.675103 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6953deba-db7d-44a8-97e9-f356fff60ff4-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:09.295382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:09.295353 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" Nov 02 22:22:09.295593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:09.295387 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk" event={"ID":"6953deba-db7d-44a8-97e9-f356fff60ff4","Type":"ContainerDied","Data":"55f647891e244d779ddac7c09201f8ceb9775c04ce5072e3a6f878311e3855b8"} Nov 02 22:22:09.295593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:09.295434 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55f647891e244d779ddac7c09201f8ceb9775c04ce5072e3a6f878311e3855b8" Nov 02 22:22:25.802734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.802694 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf"] Nov 02 22:22:25.803265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.803252 5109 cpu_manager.go:414] "containerMap: removing container" podUID="6953deba-db7d-44a8-97e9-f356fff60ff4" containerName="revoke-keys" Nov 02 22:22:25.803310 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.803266 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6953deba-db7d-44a8-97e9-f356fff60ff4" containerName="revoke-keys" Nov 02 22:22:25.803394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.803386 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="6953deba-db7d-44a8-97e9-f356fff60ff4" containerName="revoke-keys" Nov 02 22:22:25.807249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.807231 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.867428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.867395 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tls-certs\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.867428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.867436 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tmp-dir\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.867692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.867553 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-model-cache\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.867692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.867600 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-home\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.867692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.867647 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rpx\" (UniqueName: \"kubernetes.io/projected/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-kube-api-access-45rpx\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.867804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.867750 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-dshm\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.905670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.905638 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"kube-root-ca.crt\"" Nov 02 22:22:25.905670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.905638 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"default-dockercfg-bzn49\"" Nov 02 22:22:25.905670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.905648 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"distinct-w6-49a557-kserve-self-signed-certs\"" Nov 02 22:22:25.905929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.905704 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"openshift-service-ca.crt\"" Nov 02 22:22:25.909390 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.909367 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf"] Nov 02 22:22:25.979623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.978352 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tls-certs\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.979623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.978407 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tmp-dir\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.979623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.978487 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-model-cache\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.979623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.978514 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-home\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.979623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.978541 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rpx\" (UniqueName: \"kubernetes.io/projected/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-kube-api-access-45rpx\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.979623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.978723 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-dshm\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.979623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.979487 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-model-cache\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.982511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.982480 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tmp-dir\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.983115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.983075 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-home\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.986927 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.986898 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-dshm\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:25.987953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:25.987866 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tls-certs\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:26.040877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:26.040847 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rpx\" (UniqueName: \"kubernetes.io/projected/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-kube-api-access-45rpx\") pod \"distinct-w6-49a557-kserve-6bc6b5694-rr5wf\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:26.121489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:26.121408 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:26.491741 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:22:26.491652 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:22:26.580525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:26.580493 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf"] Nov 02 22:22:26.581082 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:22:26.581056 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded6c83a3_6d80_4d04_a20f_27a5e8a7e537.slice/crio-4ee8188772d6bdce6d441c3df167c921b6cf2d55a4ff0cc40080ca3698eae554 WatchSource:0}: Error finding container 4ee8188772d6bdce6d441c3df167c921b6cf2d55a4ff0cc40080ca3698eae554: Status 404 returned error can't find the container with id 4ee8188772d6bdce6d441c3df167c921b6cf2d55a4ff0cc40080ca3698eae554 Nov 02 22:22:27.457397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:27.457345 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" event={"ID":"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537","Type":"ContainerStarted","Data":"091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3"} Nov 02 22:22:27.457972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:27.457410 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" event={"ID":"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537","Type":"ContainerStarted","Data":"4ee8188772d6bdce6d441c3df167c921b6cf2d55a4ff0cc40080ca3698eae554"} Nov 02 22:22:27.569306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:27.569240 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podStartSLOduration=2.569221395 podStartE2EDuration="2.569221395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:27.568272228 +0000 UTC m=+983.833265446" watchObservedRunningTime="2026-11-02 22:22:27.569221395 +0000 UTC m=+983.834214615" Nov 02 22:22:28.132607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:28.130790 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:33.861321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.861283 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x"] Nov 02 22:22:33.865777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.865750 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.894547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.894494 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jr9\" (UniqueName: \"kubernetes.io/projected/47c2f9cc-3ba0-4891-9790-d86df9dac639-kube-api-access-f6jr9\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.894818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.894588 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c2f9cc-3ba0-4891-9790-d86df9dac639-tls-certs\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.894818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.894704 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-model-cache\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.895026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.895003 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-tmp-dir\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.895120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.895076 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-home\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.895281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.895253 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-dshm\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.922973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.922938 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x"] Nov 02 22:22:33.964491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.964455 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3224ea\"/\"distinct-w4-3224ea-kserve-self-signed-certs\"" Nov 02 22:22:33.996245 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996206 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-home\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996316 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-dshm\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996364 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jr9\" (UniqueName: \"kubernetes.io/projected/47c2f9cc-3ba0-4891-9790-d86df9dac639-kube-api-access-f6jr9\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996401 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c2f9cc-3ba0-4891-9790-d86df9dac639-tls-certs\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996443 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-model-cache\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996470 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-tmp-dir\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996730 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-home\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996851 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-tmp-dir\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.996939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.996895 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-model-cache\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.998709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.998687 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-dshm\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:33.999009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:33.998989 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c2f9cc-3ba0-4891-9790-d86df9dac639-tls-certs\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:34.066595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:34.066035 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jr9\" (UniqueName: \"kubernetes.io/projected/47c2f9cc-3ba0-4891-9790-d86df9dac639-kube-api-access-f6jr9\") pod \"distinct-w4-3224ea-kserve-5dcc76b69b-2h92x\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:34.179557 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:34.179479 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:34.594846 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:22:34.594813 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47c2f9cc_3ba0_4891_9790_d86df9dac639.slice/crio-0da61f70c4d48cf60b0df99702f7808882252c74600ce09fe8c8d6404eceadf5 WatchSource:0}: Error finding container 0da61f70c4d48cf60b0df99702f7808882252c74600ce09fe8c8d6404eceadf5: Status 404 returned error can't find the container with id 0da61f70c4d48cf60b0df99702f7808882252c74600ce09fe8c8d6404eceadf5 Nov 02 22:22:34.594977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:34.594886 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x"] Nov 02 22:22:35.513737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:35.513698 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" event={"ID":"47c2f9cc-3ba0-4891-9790-d86df9dac639","Type":"ContainerStarted","Data":"57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815"} Nov 02 22:22:35.514125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:35.513753 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" event={"ID":"47c2f9cc-3ba0-4891-9790-d86df9dac639","Type":"ContainerStarted","Data":"0da61f70c4d48cf60b0df99702f7808882252c74600ce09fe8c8d6404eceadf5"} Nov 02 22:22:35.630143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:35.629984 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podStartSLOduration=2.629965233 podStartE2EDuration="2.629965233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:35.626684848 +0000 UTC m=+991.891678066" watchObservedRunningTime="2026-11-02 22:22:35.629965233 +0000 UTC m=+991.894958452" Nov 02 22:22:36.121886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:36.121850 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:36.134851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:36.134822 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:36.179865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:36.179835 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:36.535430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:36.535395 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:22:44.180501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:44.180417 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:44.193711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:44.193678 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:44.593026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:44.592997 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:22:53.979266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:53.979224 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8"] Nov 02 22:22:53.983311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:53.983288 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.084137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.084100 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8"] Nov 02 22:22:54.117437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117397 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgq5q\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-kube-api-access-pgq5q\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.117671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117467 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.117671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117638 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.117770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117683 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.117881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117860 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.117926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117898 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.117975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117957 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242212e7-a571-4d7d-b880-c6ee0908cdf3-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.118021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.117993 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.118083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.118065 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.219608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219559 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.219819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219654 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgq5q\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-kube-api-access-pgq5q\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.219819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219679 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.219819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219708 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.219819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219727 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.219819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219802 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219825 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219857 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242212e7-a571-4d7d-b880-c6ee0908cdf3-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.219893 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220231 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.220179 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220231 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.220204 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.220325 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.220418 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.220897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.220871 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242212e7-a571-4d7d-b880-c6ee0908cdf3-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.222325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.222301 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.222714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.222693 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.277682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.277639 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.277855 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.277741 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgq5q\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-kube-api-access-pgq5q\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.298713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.298683 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:54.727415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.727382 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8"] Nov 02 22:22:54.727768 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:22:54.727746 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242212e7_a571_4d7d_b880_c6ee0908cdf3.slice/crio-2f629b1412226adf6e06d8e5b4eabeab2d226546c3cb55af073983da3f5a063d WatchSource:0}: Error finding container 2f629b1412226adf6e06d8e5b4eabeab2d226546c3cb55af073983da3f5a063d: Status 404 returned error can't find the container with id 2f629b1412226adf6e06d8e5b4eabeab2d226546c3cb55af073983da3f5a063d Nov 02 22:22:54.730249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.730218 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:22:54.730327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.730285 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:22:54.730327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:54.730315 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:22:55.657859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:55.657821 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" event={"ID":"242212e7-a571-4d7d-b880-c6ee0908cdf3","Type":"ContainerStarted","Data":"fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda"} Nov 02 22:22:55.658294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:55.657879 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" event={"ID":"242212e7-a571-4d7d-b880-c6ee0908cdf3","Type":"ContainerStarted","Data":"2f629b1412226adf6e06d8e5b4eabeab2d226546c3cb55af073983da3f5a063d"} Nov 02 22:22:55.819331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:55.819272 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podStartSLOduration=2.819257902 podStartE2EDuration="2.819257902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:55.818046157 +0000 UTC m=+1012.083039378" watchObservedRunningTime="2026-11-02 22:22:55.819257902 +0000 UTC m=+1012.084251120" Nov 02 22:22:56.299754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:56.299704 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:22:56.301052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:56.301028 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:22:56.301168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:56.301076 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 22:22:56.491656 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:22:56.491618 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:22:57.300035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:57.299995 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:22:57.300620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:57.300067 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 22:22:58.299336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:58.299302 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:22:58.299583 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:58.299357 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 22:22:59.299366 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:59.299322 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:22:59.300112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:22:59.300077 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 22:23:00.299826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:00.299793 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:23:00.300267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:00.299849 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 22:23:01.300088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:01.300049 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:23:01.300510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:01.300135 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 22:23:02.303957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:02.303924 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:23:02.304484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:02.304463 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:23:02.305250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:02.305232 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:23:02.984555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:02.984523 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv"] Nov 02 22:23:02.989052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:02.989031 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.041495 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.041447 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"kube-root-ca.crt\"" Nov 02 22:23:03.041721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.041615 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"openshift-service-ca.crt\"" Nov 02 22:23:03.042844 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.042808 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"default-dockercfg-5tbpv\"" Nov 02 22:23:03.045754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.045717 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv"] Nov 02 22:23:03.052371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.052341 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59ef92f-e353-45e2-9512-70791c54d288-tls-certs\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.052546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.052526 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lh9\" (UniqueName: \"kubernetes.io/projected/d59ef92f-e353-45e2-9512-70791c54d288-kube-api-access-v2lh9\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.052693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.052627 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-tmp-dir\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.052769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.052706 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-model-cache\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.052769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.052726 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-dshm\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.052769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.052759 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-home\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.087556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.087522 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"distinct-w1-ca85db-kserve-self-signed-certs\"" Nov 02 22:23:03.174481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.174439 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lh9\" (UniqueName: \"kubernetes.io/projected/d59ef92f-e353-45e2-9512-70791c54d288-kube-api-access-v2lh9\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.174864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.174816 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-tmp-dir\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.174999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.174901 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-model-cache\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.174999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.174935 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-dshm\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.174999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.174978 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-home\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.175165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.175037 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59ef92f-e353-45e2-9512-70791c54d288-tls-certs\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.176095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.176066 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-tmp-dir\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.176815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.176764 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-model-cache\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.176815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.176804 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-home\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.185592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.183715 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59ef92f-e353-45e2-9512-70791c54d288-tls-certs\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.186087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.186019 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-dshm\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.248423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.248353 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lh9\" (UniqueName: \"kubernetes.io/projected/d59ef92f-e353-45e2-9512-70791c54d288-kube-api-access-v2lh9\") pod \"distinct-w1-ca85db-kserve-659f84f889-qkfcv\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.302755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.302716 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:03.679720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.679687 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv"] Nov 02 22:23:03.680088 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:23:03.679895 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd59ef92f_e353_45e2_9512_70791c54d288.slice/crio-05ad927fe022e568c862236ba0aad2de50994f4d3178b4450f399bab6f5fd720 WatchSource:0}: Error finding container 05ad927fe022e568c862236ba0aad2de50994f4d3178b4450f399bab6f5fd720: Status 404 returned error can't find the container with id 05ad927fe022e568c862236ba0aad2de50994f4d3178b4450f399bab6f5fd720 Nov 02 22:23:03.716491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:03.716459 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" event={"ID":"d59ef92f-e353-45e2-9512-70791c54d288","Type":"ContainerStarted","Data":"05ad927fe022e568c862236ba0aad2de50994f4d3178b4450f399bab6f5fd720"} Nov 02 22:23:04.725216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:04.725166 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" event={"ID":"d59ef92f-e353-45e2-9512-70791c54d288","Type":"ContainerStarted","Data":"58fa74e1945c4dac3161ab3d3f8e38a8e0457459c8cbe46e7400fe9922911af6"} Nov 02 22:23:04.838543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:04.838483 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podStartSLOduration=2.838469364 podStartE2EDuration="2.838469364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:04.836307327 +0000 UTC m=+1021.101300546" watchObservedRunningTime="2026-11-02 22:23:04.838469364 +0000 UTC m=+1021.103462582" Nov 02 22:23:05.303537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:05.303498 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:13.303081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:13.303038 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:13.316116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:13.316081 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:13.801917 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:13.801886 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:23:26.491613 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:23:26.491543 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:23:40.616399 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.616360 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf"] Nov 02 22:23:40.622544 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.622519 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.720865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.720827 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf"] Nov 02 22:23:40.756792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.756747 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.756792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.756791 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205f8467-9b38-454e-b15c-b97e553bee58-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.757008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.756870 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.757069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.757050 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.757112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.757100 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205f8467-9b38-454e-b15c-b97e553bee58-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.757180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.757166 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.757218 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.757191 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlmb4\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-kube-api-access-mlmb4\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.757218 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.757213 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.757282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.757244 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.858802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.858768 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205f8467-9b38-454e-b15c-b97e553bee58-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.858828 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.858861 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmb4\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-kube-api-access-mlmb4\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.858890 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.858940 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.858987 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859118 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205f8467-9b38-454e-b15c-b97e553bee58-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859173 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859282 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859370 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859396 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859627 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205f8467-9b38-454e-b15c-b97e553bee58-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859641 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.859657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.859649 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.861876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.861854 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205f8467-9b38-454e-b15c-b97e553bee58-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.862830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.862808 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.917662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.917593 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.917874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.917852 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlmb4\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-kube-api-access-mlmb4\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:40.938433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:40.938402 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:41.372308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:41.372270 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf"] Nov 02 22:23:41.373002 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:23:41.372969 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205f8467_9b38_454e_b15c_b97e553bee58.slice/crio-48f40c352ef73465ce30de4fa1847970648fd960c40362149fd2891cee8cdec1 WatchSource:0}: Error finding container 48f40c352ef73465ce30de4fa1847970648fd960c40362149fd2891cee8cdec1: Status 404 returned error can't find the container with id 48f40c352ef73465ce30de4fa1847970648fd960c40362149fd2891cee8cdec1 Nov 02 22:23:41.375496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:41.375461 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:23:41.375618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:41.375548 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:23:41.375684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:41.375614 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:23:42.049912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:42.049872 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" event={"ID":"205f8467-9b38-454e-b15c-b97e553bee58","Type":"ContainerStarted","Data":"077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259"} Nov 02 22:23:42.050343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:42.049941 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" event={"ID":"205f8467-9b38-454e-b15c-b97e553bee58","Type":"ContainerStarted","Data":"48f40c352ef73465ce30de4fa1847970648fd960c40362149fd2891cee8cdec1"} Nov 02 22:23:42.168066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:42.168000 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podStartSLOduration=2.167983984 podStartE2EDuration="2.167983984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:42.166758013 +0000 UTC m=+1058.431751255" watchObservedRunningTime="2026-11-02 22:23:42.167983984 +0000 UTC m=+1058.432977202" Nov 02 22:23:42.948597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:42.944008 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:42.948828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:42.948747 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:23:42.948828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:42.948803 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 22:23:43.939027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:43.938986 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:23:43.939656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:43.939622 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 22:23:44.944609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:44.939113 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:23:44.944609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:44.939181 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 22:23:45.945591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:45.945028 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:23:45.945591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:45.945099 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 22:23:46.944094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:46.944049 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:23:46.944425 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:46.944392 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 22:23:47.939701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:47.939663 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:23:47.940140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:47.939731 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 22:23:48.944303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:48.944271 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:48.944832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:48.944662 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:48.945486 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:48.945467 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:23:54.091506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.091469 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm"] Nov 02 22:23:54.095674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.095654 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.174965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.174925 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175050 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175135 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175256 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175216 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175341 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175321 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7802e21-b667-4a85-8e00-78b67e4a2548-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175366 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175400 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn4f\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-kube-api-access-qxn4f\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175528 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175514 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.175593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.175546 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.197321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.197284 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm"] Nov 02 22:23:54.276384 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276342 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276392 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276475 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276531 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276611 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276695 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276739 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7802e21-b667-4a85-8e00-78b67e4a2548-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276771 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.276874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.276801 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn4f\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-kube-api-access-qxn4f\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.277118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.277090 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.277278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.277210 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.277278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.277254 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.277436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.277317 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.277529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.277502 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7802e21-b667-4a85-8e00-78b67e4a2548-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.279547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.279521 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.279799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.279782 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.339140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.339098 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.339297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.339159 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn4f\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-kube-api-access-qxn4f\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.410692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.410601 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:54.839352 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:23:54.839319 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7802e21_b667_4a85_8e00_78b67e4a2548.slice/crio-d98bfd90b81fd273870753710ae14b0f969b0a186465c656ba4308e0660e68da WatchSource:0}: Error finding container d98bfd90b81fd273870753710ae14b0f969b0a186465c656ba4308e0660e68da: Status 404 returned error can't find the container with id d98bfd90b81fd273870753710ae14b0f969b0a186465c656ba4308e0660e68da Nov 02 22:23:54.839667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.839631 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm"] Nov 02 22:23:54.842166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.842136 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:23:54.842238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.842225 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:23:54.842277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:54.842258 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:23:55.155587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:55.155471 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" event={"ID":"e7802e21-b667-4a85-8e00-78b67e4a2548","Type":"ContainerStarted","Data":"bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e"} Nov 02 22:23:55.155587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:55.155545 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" event={"ID":"e7802e21-b667-4a85-8e00-78b67e4a2548","Type":"ContainerStarted","Data":"d98bfd90b81fd273870753710ae14b0f969b0a186465c656ba4308e0660e68da"} Nov 02 22:23:55.273065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:55.273015 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podStartSLOduration=1.272997294 podStartE2EDuration="1.272997294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:55.269076247 +0000 UTC m=+1071.534069466" watchObservedRunningTime="2026-11-02 22:23:55.272997294 +0000 UTC m=+1071.537990513" Nov 02 22:23:55.411085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:55.410985 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:23:55.412916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:55.412884 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:23:55.413068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:55.412943 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 22:23:56.411669 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:56.411631 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:23:56.412176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:56.411695 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 22:23:56.492585 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:23:56.492538 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:23:57.412052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:57.412009 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:23:57.412602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:57.412094 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 22:23:58.411219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:58.411179 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:23:58.411423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:58.411277 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 22:23:59.411534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:59.411496 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:23:59.412012 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:23:59.411553 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 22:24:00.411808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:00.411768 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:24:00.412344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:00.411847 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 22:24:01.411441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:01.411408 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:24:01.411680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:01.411460 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 22:24:02.416492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:02.416460 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:24:02.416952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:02.416892 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:24:02.417673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:02.417657 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:24:10.524349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.524313 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8"] Nov 02 22:24:10.529139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.529112 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.580121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.580084 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-98a222-openshift-default-dockercfg-hhv4j\"" Nov 02 22:24:10.636609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.636547 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8"] Nov 02 22:24:10.722448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722411 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722453 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722548 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722666 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722707 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722734 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722758 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhkp\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-kube-api-access-9zhkp\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722828 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.722887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.722879 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824015 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824054 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824210 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824271 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824323 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824372 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824414 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhkp\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-kube-api-access-9zhkp\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824453 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824530 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824703 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.824761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824746 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.825009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.824959 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.825009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.825002 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.825615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.825547 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.826480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.826452 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.826916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.826898 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.882335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.882298 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:10.882604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:10.882545 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhkp\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-kube-api-access-9zhkp\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:11.144599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:11.144483 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:11.575328 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:24:11.575296 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b68edd_abe0_4c44_9f70_c8dfb4dcfcbf.slice/crio-d048289af1580b35f749f30111bd75cf44de1fcbfca712619fe9d57935ae0de5 WatchSource:0}: Error finding container d048289af1580b35f749f30111bd75cf44de1fcbfca712619fe9d57935ae0de5: Status 404 returned error can't find the container with id d048289af1580b35f749f30111bd75cf44de1fcbfca712619fe9d57935ae0de5 Nov 02 22:24:11.575707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:11.575319 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8"] Nov 02 22:24:11.577768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:11.577739 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:24:11.577830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:11.577808 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:24:11.577870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:11.577839 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:24:12.356011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:12.355962 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" event={"ID":"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf","Type":"ContainerStarted","Data":"df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d"} Nov 02 22:24:12.356210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:12.356036 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" event={"ID":"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf","Type":"ContainerStarted","Data":"d048289af1580b35f749f30111bd75cf44de1fcbfca712619fe9d57935ae0de5"} Nov 02 22:24:12.520947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:12.520888 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podStartSLOduration=2.5208735239999998 podStartE2EDuration="2.520873524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:12.517009434 +0000 UTC m=+1088.782002654" watchObservedRunningTime="2026-11-02 22:24:12.520873524 +0000 UTC m=+1088.785866742" Nov 02 22:24:13.149593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:13.149244 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:13.149593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:13.149380 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:24:13.149593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:13.149432 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 22:24:14.148728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:14.148682 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:24:14.148922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:14.148749 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 22:24:15.145595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:15.145006 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:24:15.145595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:15.145087 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 22:24:16.145596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:16.145014 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:24:16.145596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:16.145090 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 22:24:17.148594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:17.147994 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:24:17.148594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:17.148067 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 22:24:18.145336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:18.145296 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:24:18.145541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:18.145365 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 22:24:19.150109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:19.150078 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:19.150625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:19.150606 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:19.151297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:19.151278 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:24:26.491709 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:24:26.491662 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:24:39.147717 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.147681 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6"] Nov 02 22:24:39.152353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.152332 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.322867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.322829 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.322867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.322881 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb031617-7de9-4160-ad6b-762744f3c105-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.323099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.322905 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66c5\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-kube-api-access-z66c5\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.323099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.323074 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.323170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.323106 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb031617-7de9-4160-ad6b-762744f3c105-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.323209 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.323174 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.323209 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.323197 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.323297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.323278 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.323343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.323330 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.362402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.362361 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6"] Nov 02 22:24:39.424787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.424684 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.424787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.424749 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.424844 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.424882 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb031617-7de9-4160-ad6b-762744f3c105-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.424914 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66c5\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-kube-api-access-z66c5\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.424999 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425044 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb031617-7de9-4160-ad6b-762744f3c105-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425092 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425103 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425287 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425317 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425392 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425411 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.425784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.425765 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb031617-7de9-4160-ad6b-762744f3c105-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.427276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.427252 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.427506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.427487 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb031617-7de9-4160-ad6b-762744f3c105-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.483371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.483334 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.483524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.483439 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66c5\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-kube-api-access-z66c5\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:39.767115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:39.767073 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:40.208215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.208183 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6"] Nov 02 22:24:40.208598 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:24:40.208273 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb031617_7de9_4160_ad6b_762744f3c105.slice/crio-4c2b9d47b90716395030bd769c12233ccae8d341948ca1b6b43630a6df5a4aad WatchSource:0}: Error finding container 4c2b9d47b90716395030bd769c12233ccae8d341948ca1b6b43630a6df5a4aad: Status 404 returned error can't find the container with id 4c2b9d47b90716395030bd769c12233ccae8d341948ca1b6b43630a6df5a4aad Nov 02 22:24:40.210867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.210834 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:24:40.210960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.210897 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:24:40.210960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.210923 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:24:40.612756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.612718 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" event={"ID":"eb031617-7de9-4160-ad6b-762744f3c105","Type":"ContainerStarted","Data":"c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882"} Nov 02 22:24:40.612937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.612774 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" event={"ID":"eb031617-7de9-4160-ad6b-762744f3c105","Type":"ContainerStarted","Data":"4c2b9d47b90716395030bd769c12233ccae8d341948ca1b6b43630a6df5a4aad"} Nov 02 22:24:40.767685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.767650 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:40.773691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:40.773644 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podStartSLOduration=1.773631097 podStartE2EDuration="1.773631097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:40.7735553 +0000 UTC m=+1117.038548520" watchObservedRunningTime="2026-11-02 22:24:40.773631097 +0000 UTC m=+1117.038624310" Nov 02 22:24:41.769505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:41.769473 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:24:41.769952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:41.769525 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 22:24:42.768303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:42.768265 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:24:42.768474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:42.768324 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 22:24:43.767968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:43.767930 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:24:43.768401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:43.768011 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 22:24:44.767911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:44.767872 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:24:44.768111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:44.767934 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 22:24:45.768388 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:45.768352 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:24:45.768853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:45.768408 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 22:24:46.768491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:46.768451 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:24:46.768952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:46.768523 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 22:24:47.773846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:47.773793 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:47.774362 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:47.774223 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:47.775250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:47.775226 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:24:51.368997 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.368958 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp"] Nov 02 22:24:51.373395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.373375 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.406697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.406662 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-tmp-dir\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.406917 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.406816 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-home\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.406917 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.406853 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jfn\" (UniqueName: \"kubernetes.io/projected/8c658010-2eae-42cf-8a70-f08f3014976c-kube-api-access-k4jfn\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.407048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.406962 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-dshm\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.407048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.406988 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-model-cache\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.407048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.407005 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c658010-2eae-42cf-8a70-f08f3014976c-tls-certs\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.470750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.470709 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3224ea\"/\"distinct-2-w4-3224ea-kserve-self-signed-certs\"" Nov 02 22:24:51.474921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.474893 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp"] Nov 02 22:24:51.548592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.520510 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-home\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.548943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.548897 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jfn\" (UniqueName: \"kubernetes.io/projected/8c658010-2eae-42cf-8a70-f08f3014976c-kube-api-access-k4jfn\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.549684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.549659 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-dshm\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.550297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.550275 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-model-cache\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.550452 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.550418 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c658010-2eae-42cf-8a70-f08f3014976c-tls-certs\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.561407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.561351 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-tmp-dir\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.562296 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.562254 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-tmp-dir\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.562484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.543890 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-home\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.562935 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.562901 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-model-cache\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.584532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.567956 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-dshm\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.595026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.594985 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c658010-2eae-42cf-8a70-f08f3014976c-tls-certs\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.607634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.607600 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jfn\" (UniqueName: \"kubernetes.io/projected/8c658010-2eae-42cf-8a70-f08f3014976c-kube-api-access-k4jfn\") pod \"distinct-2-w4-3224ea-kserve-749768b767-99llp\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:51.687642 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:51.687536 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:52.147126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:52.147097 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp"] Nov 02 22:24:52.147253 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:24:52.147225 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c658010_2eae_42cf_8a70_f08f3014976c.slice/crio-8f1ecc54513ff4987125318d3650e88231976ed52a177972cfbc6e7f122790e7 WatchSource:0}: Error finding container 8f1ecc54513ff4987125318d3650e88231976ed52a177972cfbc6e7f122790e7: Status 404 returned error can't find the container with id 8f1ecc54513ff4987125318d3650e88231976ed52a177972cfbc6e7f122790e7 Nov 02 22:24:52.732445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:52.732400 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" event={"ID":"8c658010-2eae-42cf-8a70-f08f3014976c","Type":"ContainerStarted","Data":"824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2"} Nov 02 22:24:52.732445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:52.732450 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" event={"ID":"8c658010-2eae-42cf-8a70-f08f3014976c","Type":"ContainerStarted","Data":"8f1ecc54513ff4987125318d3650e88231976ed52a177972cfbc6e7f122790e7"} Nov 02 22:24:52.845370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:52.845314 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podStartSLOduration=1.845296045 podStartE2EDuration="1.845296045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:52.843821837 +0000 UTC m=+1129.108815068" watchObservedRunningTime="2026-11-02 22:24:52.845296045 +0000 UTC m=+1129.110289264" Nov 02 22:24:53.688676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:53.688637 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:24:54.551528 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.551488 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g"] Nov 02 22:24:54.556048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.556027 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.616925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.614075 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g"] Nov 02 22:24:54.624890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.624858 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:54.652474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.652433 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:24:54.739622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.729607 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbpvk\" (UniqueName: \"kubernetes.io/projected/cdf0d71e-3de9-4424-84df-8dc916ee2196-kube-api-access-nbpvk\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.739622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.729772 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.739622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.729848 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.841961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.841871 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpvk\" (UniqueName: \"kubernetes.io/projected/cdf0d71e-3de9-4424-84df-8dc916ee2196-kube-api-access-nbpvk\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.841961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.841952 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.842179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.841991 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.842800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.842768 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.844693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.844669 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:54.900502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:54.900471 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpvk\" (UniqueName: \"kubernetes.io/projected/cdf0d71e-3de9-4424-84df-8dc916ee2196-kube-api-access-nbpvk\") pod \"maas-api-revoke-sub-cbec0d9b-p9r9g\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:55.170798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:55.170715 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:55.547453 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:24:55.547418 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf0d71e_3de9_4424_84df_8dc916ee2196.slice/crio-4282d4c7dcd760ec41eae4b9104d7859991dc255263b75533762c0234d97abae WatchSource:0}: Error finding container 4282d4c7dcd760ec41eae4b9104d7859991dc255263b75533762c0234d97abae: Status 404 returned error can't find the container with id 4282d4c7dcd760ec41eae4b9104d7859991dc255263b75533762c0234d97abae Nov 02 22:24:55.547724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:55.547698 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g"] Nov 02 22:24:55.768288 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:55.768253 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" event={"ID":"cdf0d71e-3de9-4424-84df-8dc916ee2196","Type":"ContainerStarted","Data":"4282d4c7dcd760ec41eae4b9104d7859991dc255263b75533762c0234d97abae"} Nov 02 22:24:56.492370 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:24:56.492333 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:24:56.862315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:56.862277 5109 generic.go:356] "Generic (PLEG): container finished" podID="cdf0d71e-3de9-4424-84df-8dc916ee2196" containerID="b664da216b4a4a4aa57645751cee8f4408692ac3927ad3b5990f67a7b8f07813" exitCode=0 Nov 02 22:24:56.863169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:56.863137 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" event={"ID":"cdf0d71e-3de9-4424-84df-8dc916ee2196","Type":"ContainerDied","Data":"b664da216b4a4a4aa57645751cee8f4408692ac3927ad3b5990f67a7b8f07813"} Nov 02 22:24:58.038542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.038513 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:58.091945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.091900 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf0d71e-3de9-4424-84df-8dc916ee2196-kube-api-access-nbpvk\" (UniqueName: \"kubernetes.io/projected/cdf0d71e-3de9-4424-84df-8dc916ee2196-kube-api-access-nbpvk\") pod \"cdf0d71e-3de9-4424-84df-8dc916ee2196\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " Nov 02 22:24:58.092122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.091968 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-xks-ca\") pod \"cdf0d71e-3de9-4424-84df-8dc916ee2196\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " Nov 02 22:24:58.092212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.092164 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-openshift-ca\") pod \"cdf0d71e-3de9-4424-84df-8dc916ee2196\" (UID: \"cdf0d71e-3de9-4424-84df-8dc916ee2196\") " Nov 02 22:24:58.092515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.092490 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-openshift-ca" pod "cdf0d71e-3de9-4424-84df-8dc916ee2196" (UID: "cdf0d71e-3de9-4424-84df-8dc916ee2196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:58.092756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.092739 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:58.094312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.094290 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-xks-ca" pod "cdf0d71e-3de9-4424-84df-8dc916ee2196" (UID: "cdf0d71e-3de9-4424-84df-8dc916ee2196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:58.094402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.094344 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf0d71e-3de9-4424-84df-8dc916ee2196-kube-api-access-nbpvk" pod "cdf0d71e-3de9-4424-84df-8dc916ee2196" (UID: "cdf0d71e-3de9-4424-84df-8dc916ee2196"). InnerVolumeSpecName "kube-api-access-nbpvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:58.194238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.194144 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbpvk\" (UniqueName: \"kubernetes.io/projected/cdf0d71e-3de9-4424-84df-8dc916ee2196-kube-api-access-nbpvk\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:58.194238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.194184 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf0d71e-3de9-4424-84df-8dc916ee2196-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:58.883240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.883205 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" Nov 02 22:24:58.883240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.883225 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g" event={"ID":"cdf0d71e-3de9-4424-84df-8dc916ee2196","Type":"ContainerDied","Data":"4282d4c7dcd760ec41eae4b9104d7859991dc255263b75533762c0234d97abae"} Nov 02 22:24:58.883506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:24:58.883303 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4282d4c7dcd760ec41eae4b9104d7859991dc255263b75533762c0234d97abae" Nov 02 22:25:01.687914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:01.687879 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:25:01.700477 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:01.700446 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:25:01.920945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:01.920907 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:25:08.059322 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:08.059283 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9"] Nov 02 22:25:08.104854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:08.104810 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b69049a-w4ns9"] Nov 02 22:25:09.892510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:09.892464 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e490a7bf-cae0-4d82-81a8-b80327649fb3" path="/var/lib/kubelet/pods/e490a7bf-cae0-4d82-81a8-b80327649fb3/volumes" Nov 02 22:25:16.052190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:16.052155 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd"] Nov 02 22:25:16.098701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:16.098665 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1530f7-fbkzd"] Nov 02 22:25:17.894108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:17.894077 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d4af33-9f66-49f1-9bdd-f3500fad3569" path="/var/lib/kubelet/pods/04d4af33-9f66-49f1-9bdd-f3500fad3569/volumes" Nov 02 22:25:23.050364 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:23.050321 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv"] Nov 02 22:25:23.096534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:23.096495 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f341310-cs6bv"] Nov 02 22:25:23.893095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:23.893063 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50" path="/var/lib/kubelet/pods/ed01dc6a-51cc-4d4a-87d6-5b3a0e71aa50/volumes" Nov 02 22:25:25.078084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.078050 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs"] Nov 02 22:25:25.078842 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.078824 5109 cpu_manager.go:414] "containerMap: removing container" podUID="cdf0d71e-3de9-4424-84df-8dc916ee2196" containerName="revoke-keys" Nov 02 22:25:25.078907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.078844 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf0d71e-3de9-4424-84df-8dc916ee2196" containerName="revoke-keys" Nov 02 22:25:25.078974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.078964 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf0d71e-3de9-4424-84df-8dc916ee2196" containerName="revoke-keys" Nov 02 22:25:25.083641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.083622 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.125189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.125152 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf"] Nov 02 22:25:25.129696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.129667 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.177174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.177137 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs"] Nov 02 22:25:25.230587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.230539 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf"] Nov 02 22:25:25.260998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.260960 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261177 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261005 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261177 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261078 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261192 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261221 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261278 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8c4h\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-kube-api-access-x8c4h\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261312 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261352 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261379 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261408 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqm7\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-kube-api-access-cdqm7\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261432 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261452 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c12691b-81b3-4967-837c-f32390a02cbb-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261621 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261669 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261783 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1aa813-c5de-4562-91f7-ea2c64086c30-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.261839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261820 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261846 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c12691b-81b3-4967-837c-f32390a02cbb-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.261981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.261938 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363017 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363057 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363100 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1aa813-c5de-4562-91f7-ea2c64086c30-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363124 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363140 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c12691b-81b3-4967-837c-f32390a02cbb-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363159 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363201 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363262 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363299 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363396 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363424 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363463 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8c4h\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-kube-api-access-x8c4h\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363524 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363527 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363618 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363630 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363676 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363705 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqm7\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-kube-api-access-cdqm7\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363761 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363790 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c12691b-81b3-4967-837c-f32390a02cbb-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.363826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363814 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.364372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363860 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.364372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.363872 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.364372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.364168 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c12691b-81b3-4967-837c-f32390a02cbb-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.364604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.364554 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.364737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.364709 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.364801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.364709 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1aa813-c5de-4562-91f7-ea2c64086c30-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.365053 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.365033 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.366523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.366494 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.366523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.366511 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.366815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.366796 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c12691b-81b3-4967-837c-f32390a02cbb-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.367122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.367100 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.421898 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.421865 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.422058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.421960 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.422058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.422045 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqm7\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-kube-api-access-cdqm7\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.422540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.422518 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8c4h\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-kube-api-access-x8c4h\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.444307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.444273 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:25.697366 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.697270 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:25.881715 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:25.881680 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c12691b_81b3_4967_837c_f32390a02cbb.slice/crio-d8350510176c6be32941651a146c6876374001d9dd9a06f439560a3b1b9b9ae6 WatchSource:0}: Error finding container d8350510176c6be32941651a146c6876374001d9dd9a06f439560a3b1b9b9ae6: Status 404 returned error can't find the container with id d8350510176c6be32941651a146c6876374001d9dd9a06f439560a3b1b9b9ae6 Nov 02 22:25:25.882456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.882424 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf"] Nov 02 22:25:25.884637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.884595 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:25.884759 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.884702 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:25.884759 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:25.884743 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:26.082260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.082231 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs"] Nov 02 22:25:26.084438 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:26.084411 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1aa813_c5de_4562_91f7_ea2c64086c30.slice/crio-b13a2e149e277483f0607ede8c00750d4bb2f8cd5f2116260e7135967ba58f18 WatchSource:0}: Error finding container b13a2e149e277483f0607ede8c00750d4bb2f8cd5f2116260e7135967ba58f18: Status 404 returned error can't find the container with id b13a2e149e277483f0607ede8c00750d4bb2f8cd5f2116260e7135967ba58f18 Nov 02 22:25:26.088625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.088098 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:26.088625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.088194 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:26.088625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.088241 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:26.103898 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.103853 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng"] Nov 02 22:25:26.109600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.109537 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.172754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.170512 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng"] Nov 02 22:25:26.277344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.277304 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb24859c-e64b-4f72-b230-c85c1b09c596-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.277554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.277365 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.277554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.277485 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.277748 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.277722 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.278031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.277819 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.278031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.277872 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtw5m\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-kube-api-access-wtw5m\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.278031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.277977 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.278031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.278007 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.278345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.278193 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.367638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.364142 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" event={"ID":"3b1aa813-c5de-4562-91f7-ea2c64086c30","Type":"ContainerStarted","Data":"f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e"} Nov 02 22:25:26.367638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.364218 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" event={"ID":"3b1aa813-c5de-4562-91f7-ea2c64086c30","Type":"ContainerStarted","Data":"b13a2e149e277483f0607ede8c00750d4bb2f8cd5f2116260e7135967ba58f18"} Nov 02 22:25:26.367638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.366224 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" event={"ID":"5c12691b-81b3-4967-837c-f32390a02cbb","Type":"ContainerStarted","Data":"b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0"} Nov 02 22:25:26.367638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.366272 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" event={"ID":"5c12691b-81b3-4967-837c-f32390a02cbb","Type":"ContainerStarted","Data":"d8350510176c6be32941651a146c6876374001d9dd9a06f439560a3b1b9b9ae6"} Nov 02 22:25:26.379292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379249 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379328 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379376 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtw5m\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-kube-api-access-wtw5m\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379438 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379463 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379531 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379595 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb24859c-e64b-4f72-b230-c85c1b09c596-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379633 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379670 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.379750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.379730 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.380048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.380026 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.380317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.380291 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.380406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.380341 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.380742 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.380708 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb24859c-e64b-4f72-b230-c85c1b09c596-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.383308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.383277 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.383694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.383666 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.438312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.438280 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.438488 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.438464 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtw5m\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-kube-api-access-wtw5m\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:26.444799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.444769 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:26.446277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.446250 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:26.446412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.446303 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:26.482282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.482217 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podStartSLOduration=1.4821971569999999 podStartE2EDuration="1.482197157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:26.478844579 +0000 UTC m=+1162.743837802" watchObservedRunningTime="2026-11-02 22:25:26.482197157 +0000 UTC m=+1162.747190377" Nov 02 22:25:26.497082 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:25:26.497026 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:25:26.599343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.599224 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podStartSLOduration=1.5992035759999998 podStartE2EDuration="1.599203576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:26.594544437 +0000 UTC m=+1162.859537649" watchObservedRunningTime="2026-11-02 22:25:26.599203576 +0000 UTC m=+1162.864196796" Nov 02 22:25:26.697649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.697613 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:26.738093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:26.738052 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:27.181668 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:27.181636 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb24859c_e64b_4f72_b230_c85c1b09c596.slice/crio-8d3f56d56282cda7752ea63a0e6390271d9e7de2c3820c9be5fb21d6e808d29c WatchSource:0}: Error finding container 8d3f56d56282cda7752ea63a0e6390271d9e7de2c3820c9be5fb21d6e808d29c: Status 404 returned error can't find the container with id 8d3f56d56282cda7752ea63a0e6390271d9e7de2c3820c9be5fb21d6e808d29c Nov 02 22:25:27.182828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.182803 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng"] Nov 02 22:25:27.184426 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.184395 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:27.184504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.184463 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:27.184504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.184491 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:27.392208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.392144 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" event={"ID":"bb24859c-e64b-4f72-b230-c85c1b09c596","Type":"ContainerStarted","Data":"b846111df9472d34d34163dcd18be102ba5d46f53e8fea7b43d4585a1d26504f"} Nov 02 22:25:27.392788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.392271 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" event={"ID":"bb24859c-e64b-4f72-b230-c85c1b09c596","Type":"ContainerStarted","Data":"8d3f56d56282cda7752ea63a0e6390271d9e7de2c3820c9be5fb21d6e808d29c"} Nov 02 22:25:27.448668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.446079 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:27.448668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.446142 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:27.514505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.514457 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podStartSLOduration=1.514440947 podStartE2EDuration="1.514440947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:27.510471696 +0000 UTC m=+1163.775464915" watchObservedRunningTime="2026-11-02 22:25:27.514440947 +0000 UTC m=+1163.779434166" Nov 02 22:25:27.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.700484 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:25:27.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.700588 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 22:25:27.747596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:27.746689 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:28.448597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:28.448075 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:28.448597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:28.448140 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:28.701251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:28.701147 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:25:28.701538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:28.701491 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 22:25:28.760594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:28.760035 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:25:28.760594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:28.760100 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:25:29.445593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:29.445012 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:29.445593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:29.445075 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:29.717654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:29.700001 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:25:29.717654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:29.700110 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 22:25:29.741605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:29.741005 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:25:29.741605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:29.741069 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:25:30.479595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:30.479400 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:30.479595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:30.479463 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:30.700596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:30.700223 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:25:30.700596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:30.700321 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 22:25:30.744655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:30.742090 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:25:30.744655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:30.742170 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:25:31.475596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:31.474732 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:31.475596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:31.474803 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:31.699408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:31.698967 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:25:31.699408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:31.699028 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 22:25:31.744653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:31.743691 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:25:31.744653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:31.743760 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:25:32.458595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:32.450045 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:32.458595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:32.450119 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:32.698055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:32.698016 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:25:32.698263 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:32.698080 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 22:25:32.739100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:32.739002 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:25:32.739386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:32.739354 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:25:33.445026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.444981 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:25:33.446127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.446087 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 22:25:33.485490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.485442 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj"] Nov 02 22:25:33.492937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.492903 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.545057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.544991 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:33.567954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.567897 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj"] Nov 02 22:25:33.589536 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.589492 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:25:33.596002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.595962 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xps2\" (UniqueName: \"kubernetes.io/projected/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-kube-api-access-2xps2\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.596207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.596191 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.596273 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.596221 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.697974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.697857 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.697974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.697905 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.698232 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.698019 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xps2\" (UniqueName: \"kubernetes.io/projected/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-kube-api-access-2xps2\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.698688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.698663 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.698894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.698868 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:25:33.698968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.698921 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 22:25:33.701358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.701332 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.750355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.750318 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:25:33.750650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.750620 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:25:33.760546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.760509 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xps2\" (UniqueName: \"kubernetes.io/projected/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-kube-api-access-2xps2\") pod \"maas-api-revoke-sub-b93686c3-972dj\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:33.810215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:33.810178 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:34.351769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.351732 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj"] Nov 02 22:25:34.355439 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:34.355403 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769661e5_9a2f_4e95_b3c2_ade0dfe3a547.slice/crio-eebb8407c436c251ef58fd6f972cd42c7f7795316ff08e0171ec5ee3e3c2ebd8 WatchSource:0}: Error finding container eebb8407c436c251ef58fd6f972cd42c7f7795316ff08e0171ec5ee3e3c2ebd8: Status 404 returned error can't find the container with id eebb8407c436c251ef58fd6f972cd42c7f7795316ff08e0171ec5ee3e3c2ebd8 Nov 02 22:25:34.479185 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.477153 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:34.479185 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.478771 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:34.499814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.488003 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:25:34.594611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.593619 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" event={"ID":"769661e5-9a2f-4e95-b3c2-ade0dfe3a547","Type":"ContainerStarted","Data":"eebb8407c436c251ef58fd6f972cd42c7f7795316ff08e0171ec5ee3e3c2ebd8"} Nov 02 22:25:34.705651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.705231 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:34.707635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.705966 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:34.707635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.707047 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:25:34.739024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.738694 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:25:34.739024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:34.738776 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:25:35.667594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:35.664003 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" event={"ID":"769661e5-9a2f-4e95-b3c2-ade0dfe3a547","Type":"ContainerStarted","Data":"562345fd491759a47236d0c0c0d4d0e583e0e276252f8c69ba1211c0a1d9a395"} Nov 02 22:25:36.533281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.531686 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:25:36.585780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.585732 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:25:36.746459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.745765 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:25:36.746459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.745838 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:25:36.874745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.806703 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:25:36.886615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.881427 5109 generic.go:356] "Generic (PLEG): container finished" podID="769661e5-9a2f-4e95-b3c2-ade0dfe3a547" containerID="562345fd491759a47236d0c0c0d4d0e583e0e276252f8c69ba1211c0a1d9a395" exitCode=0 Nov 02 22:25:36.901708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.899714 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" event={"ID":"769661e5-9a2f-4e95-b3c2-ade0dfe3a547","Type":"ContainerDied","Data":"562345fd491759a47236d0c0c0d4d0e583e0e276252f8c69ba1211c0a1d9a395"} Nov 02 22:25:36.976594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:36.975786 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:25:37.145162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:37.144691 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Nov 02 22:25:37.346592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:37.342681 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:25:37.786785 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:37.786739 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:25:37.787284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:37.787248 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:25:38.115002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.114974 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:38.132245 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.132206 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-kube-api-access-2xps2\" (UniqueName: \"kubernetes.io/projected/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-kube-api-access-2xps2\") pod \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " Nov 02 22:25:38.132410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.132293 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-xks-ca\") pod \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " Nov 02 22:25:38.132410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.132393 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-openshift-ca\") pod \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\" (UID: \"769661e5-9a2f-4e95-b3c2-ade0dfe3a547\") " Nov 02 22:25:38.132879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.132826 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-openshift-ca" pod "769661e5-9a2f-4e95-b3c2-ade0dfe3a547" (UID: "769661e5-9a2f-4e95-b3c2-ade0dfe3a547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:38.133475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.133444 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:38.134805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.134775 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-kube-api-access-2xps2" pod "769661e5-9a2f-4e95-b3c2-ade0dfe3a547" (UID: "769661e5-9a2f-4e95-b3c2-ade0dfe3a547"). InnerVolumeSpecName "kube-api-access-2xps2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:38.135036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.135009 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-xks-ca" pod "769661e5-9a2f-4e95-b3c2-ade0dfe3a547" (UID: "769661e5-9a2f-4e95-b3c2-ade0dfe3a547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:38.250221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.250152 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xps2\" (UniqueName: \"kubernetes.io/projected/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-kube-api-access-2xps2\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:38.250221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.250192 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769661e5-9a2f-4e95-b3c2-ade0dfe3a547-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:38.384283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.384183 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:38.385379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.385349 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:38.386493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.386466 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:25:38.919809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.919767 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" event={"ID":"769661e5-9a2f-4e95-b3c2-ade0dfe3a547","Type":"ContainerDied","Data":"eebb8407c436c251ef58fd6f972cd42c7f7795316ff08e0171ec5ee3e3c2ebd8"} Nov 02 22:25:38.919809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.919794 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj" Nov 02 22:25:38.920331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:38.919829 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eebb8407c436c251ef58fd6f972cd42c7f7795316ff08e0171ec5ee3e3c2ebd8" Nov 02 22:25:39.013353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.011741 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88"] Nov 02 22:25:39.013353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.013270 5109 cpu_manager.go:414] "containerMap: removing container" podUID="769661e5-9a2f-4e95-b3c2-ade0dfe3a547" containerName="revoke-keys" Nov 02 22:25:39.013353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.013293 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769661e5-9a2f-4e95-b3c2-ade0dfe3a547" containerName="revoke-keys" Nov 02 22:25:39.013790 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.013695 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="769661e5-9a2f-4e95-b3c2-ade0dfe3a547" containerName="revoke-keys" Nov 02 22:25:39.021173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.021146 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.062209 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.062178 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvw7k\" (UniqueName: \"kubernetes.io/projected/218b52b1-048e-455a-b688-b8dd6a0fa2c8-kube-api-access-lvw7k\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.062414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.062235 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.062414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.062391 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.068677 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.068644 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88"] Nov 02 22:25:39.070473 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.070445 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:39.070920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.070889 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:25:39.163747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.163701 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.163996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.163922 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvw7k\" (UniqueName: \"kubernetes.io/projected/218b52b1-048e-455a-b688-b8dd6a0fa2c8-kube-api-access-lvw7k\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.163996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.163988 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.164554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.164525 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.166892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.166866 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.222981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.222900 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvw7k\" (UniqueName: \"kubernetes.io/projected/218b52b1-048e-455a-b688-b8dd6a0fa2c8-kube-api-access-lvw7k\") pod \"maas-api-revoke-sub-7e7208d9-w7g88\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.337122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.337087 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:39.714798 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:39.714763 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218b52b1_048e_455a_b688_b8dd6a0fa2c8.slice/crio-06d70c7eac789e27111cf1d475ddbcbb0c7ee416b5392f1b67a7c8bb69ab9f0d WatchSource:0}: Error finding container 06d70c7eac789e27111cf1d475ddbcbb0c7ee416b5392f1b67a7c8bb69ab9f0d: Status 404 returned error can't find the container with id 06d70c7eac789e27111cf1d475ddbcbb0c7ee416b5392f1b67a7c8bb69ab9f0d Nov 02 22:25:39.715085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.715068 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88"] Nov 02 22:25:39.931339 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:39.931307 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" event={"ID":"218b52b1-048e-455a-b688-b8dd6a0fa2c8","Type":"ContainerStarted","Data":"06d70c7eac789e27111cf1d475ddbcbb0c7ee416b5392f1b67a7c8bb69ab9f0d"} Nov 02 22:25:40.941511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:40.941411 5109 generic.go:356] "Generic (PLEG): container finished" podID="218b52b1-048e-455a-b688-b8dd6a0fa2c8" containerID="874f4bf2ff1060912eaec8cf373a03cd14aab009b2a5efa9a27b3defa5caac41" exitCode=0 Nov 02 22:25:40.941511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:40.941499 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" event={"ID":"218b52b1-048e-455a-b688-b8dd6a0fa2c8","Type":"ContainerDied","Data":"874f4bf2ff1060912eaec8cf373a03cd14aab009b2a5efa9a27b3defa5caac41"} Nov 02 22:25:42.054984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.052644 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5"] Nov 02 22:25:42.097521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.097488 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba68aa3-n4rv5"] Nov 02 22:25:42.113649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.113618 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:42.205251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.205167 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218b52b1-048e-455a-b688-b8dd6a0fa2c8-kube-api-access-lvw7k\" (UniqueName: \"kubernetes.io/projected/218b52b1-048e-455a-b688-b8dd6a0fa2c8-kube-api-access-lvw7k\") pod \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " Nov 02 22:25:42.205414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.205279 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-openshift-ca\") pod \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " Nov 02 22:25:42.205414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.205324 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-xks-ca\") pod \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\" (UID: \"218b52b1-048e-455a-b688-b8dd6a0fa2c8\") " Nov 02 22:25:42.205744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.205717 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-openshift-ca" pod "218b52b1-048e-455a-b688-b8dd6a0fa2c8" (UID: "218b52b1-048e-455a-b688-b8dd6a0fa2c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:42.205982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.205926 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:42.207520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.207491 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-xks-ca" pod "218b52b1-048e-455a-b688-b8dd6a0fa2c8" (UID: "218b52b1-048e-455a-b688-b8dd6a0fa2c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:42.207652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.207552 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218b52b1-048e-455a-b688-b8dd6a0fa2c8-kube-api-access-lvw7k" pod "218b52b1-048e-455a-b688-b8dd6a0fa2c8" (UID: "218b52b1-048e-455a-b688-b8dd6a0fa2c8"). InnerVolumeSpecName "kube-api-access-lvw7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:42.307387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.307342 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218b52b1-048e-455a-b688-b8dd6a0fa2c8-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:42.307633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.307400 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvw7k\" (UniqueName: \"kubernetes.io/projected/218b52b1-048e-455a-b688-b8dd6a0fa2c8-kube-api-access-lvw7k\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:42.957629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.957589 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" event={"ID":"218b52b1-048e-455a-b688-b8dd6a0fa2c8","Type":"ContainerDied","Data":"06d70c7eac789e27111cf1d475ddbcbb0c7ee416b5392f1b67a7c8bb69ab9f0d"} Nov 02 22:25:42.957821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.957635 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88" Nov 02 22:25:42.957821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:42.957640 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d70c7eac789e27111cf1d475ddbcbb0c7ee416b5392f1b67a7c8bb69ab9f0d" Nov 02 22:25:43.066856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:43.066821 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j"] Nov 02 22:25:43.070745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:43.070720 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0c0451-zzq7j"] Nov 02 22:25:43.894519 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:43.894484 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9e3866-53e0-4efe-949b-9e3fea7c7555" path="/var/lib/kubelet/pods/5f9e3866-53e0-4efe-949b-9e3fea7c7555/volumes" Nov 02 22:25:43.894927 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:43.894912 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1868ed-4357-43a6-8c81-7ad7fdb356f4" path="/var/lib/kubelet/pods/8c1868ed-4357-43a6-8c81-7ad7fdb356f4/volumes" Nov 02 22:25:44.223199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.223119 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d"] Nov 02 22:25:44.223850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.223831 5109 cpu_manager.go:414] "containerMap: removing container" podUID="218b52b1-048e-455a-b688-b8dd6a0fa2c8" containerName="revoke-keys" Nov 02 22:25:44.223912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.223852 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218b52b1-048e-455a-b688-b8dd6a0fa2c8" containerName="revoke-keys" Nov 02 22:25:44.224003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.223994 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="218b52b1-048e-455a-b688-b8dd6a0fa2c8" containerName="revoke-keys" Nov 02 22:25:44.228851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.228830 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.283651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.283617 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d"] Nov 02 22:25:44.324475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.324443 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:44.324475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.324458 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:25:44.333534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.333501 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.333744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.333641 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4687\" (UniqueName: \"kubernetes.io/projected/c66635b2-e0f2-4c30-8652-7fe13d667d27-kube-api-access-j4687\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.333789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.333778 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.434869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.434829 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.435089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.435026 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.435168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.435117 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4687\" (UniqueName: \"kubernetes.io/projected/c66635b2-e0f2-4c30-8652-7fe13d667d27-kube-api-access-j4687\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.435618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.435585 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.438143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.438117 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.493436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.493353 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4687\" (UniqueName: \"kubernetes.io/projected/c66635b2-e0f2-4c30-8652-7fe13d667d27-kube-api-access-j4687\") pod \"maas-api-revoke-sub-8704f258-lrl5d\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.544909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.544877 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:44.922812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.922777 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d"] Nov 02 22:25:44.924026 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:44.923997 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66635b2_e0f2_4c30_8652_7fe13d667d27.slice/crio-49fe4a26c0f3e3ccd289da0fb568787898f4f29f30b4a65cc3cf10e900d6e0bd WatchSource:0}: Error finding container 49fe4a26c0f3e3ccd289da0fb568787898f4f29f30b4a65cc3cf10e900d6e0bd: Status 404 returned error can't find the container with id 49fe4a26c0f3e3ccd289da0fb568787898f4f29f30b4a65cc3cf10e900d6e0bd Nov 02 22:25:44.975012 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:44.974970 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" event={"ID":"c66635b2-e0f2-4c30-8652-7fe13d667d27","Type":"ContainerStarted","Data":"49fe4a26c0f3e3ccd289da0fb568787898f4f29f30b4a65cc3cf10e900d6e0bd"} Nov 02 22:25:45.984584 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:45.984530 5109 generic.go:356] "Generic (PLEG): container finished" podID="c66635b2-e0f2-4c30-8652-7fe13d667d27" containerID="263aae860fda7a73907baca01fe72bc0195c67167a9f3f52f8bc28d481d3292d" exitCode=0 Nov 02 22:25:45.985027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:45.984620 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" event={"ID":"c66635b2-e0f2-4c30-8652-7fe13d667d27","Type":"ContainerDied","Data":"263aae860fda7a73907baca01fe72bc0195c67167a9f3f52f8bc28d481d3292d"} Nov 02 22:25:47.158182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.158151 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:47.280284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.280235 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-openshift-ca\") pod \"c66635b2-e0f2-4c30-8652-7fe13d667d27\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " Nov 02 22:25:47.280451 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.280378 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66635b2-e0f2-4c30-8652-7fe13d667d27-kube-api-access-j4687\" (UniqueName: \"kubernetes.io/projected/c66635b2-e0f2-4c30-8652-7fe13d667d27-kube-api-access-j4687\") pod \"c66635b2-e0f2-4c30-8652-7fe13d667d27\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " Nov 02 22:25:47.280451 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.280429 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-xks-ca\") pod \"c66635b2-e0f2-4c30-8652-7fe13d667d27\" (UID: \"c66635b2-e0f2-4c30-8652-7fe13d667d27\") " Nov 02 22:25:47.280611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.280552 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-openshift-ca" pod "c66635b2-e0f2-4c30-8652-7fe13d667d27" (UID: "c66635b2-e0f2-4c30-8652-7fe13d667d27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:47.281016 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.280989 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:47.283252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.283222 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-xks-ca" pod "c66635b2-e0f2-4c30-8652-7fe13d667d27" (UID: "c66635b2-e0f2-4c30-8652-7fe13d667d27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:47.283252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.283237 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66635b2-e0f2-4c30-8652-7fe13d667d27-kube-api-access-j4687" pod "c66635b2-e0f2-4c30-8652-7fe13d667d27" (UID: "c66635b2-e0f2-4c30-8652-7fe13d667d27"). InnerVolumeSpecName "kube-api-access-j4687". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:47.382444 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.382402 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4687\" (UniqueName: \"kubernetes.io/projected/c66635b2-e0f2-4c30-8652-7fe13d667d27-kube-api-access-j4687\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:47.382444 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:47.382439 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66635b2-e0f2-4c30-8652-7fe13d667d27-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:48.001748 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:48.001720 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" Nov 02 22:25:48.001941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:48.001714 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d" event={"ID":"c66635b2-e0f2-4c30-8652-7fe13d667d27","Type":"ContainerDied","Data":"49fe4a26c0f3e3ccd289da0fb568787898f4f29f30b4a65cc3cf10e900d6e0bd"} Nov 02 22:25:48.001941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:48.001849 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49fe4a26c0f3e3ccd289da0fb568787898f4f29f30b4a65cc3cf10e900d6e0bd" Nov 02 22:25:51.050309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:51.050278 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb"] Nov 02 22:25:51.096402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:51.096363 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-410664e3-x5pkb"] Nov 02 22:25:51.895121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:51.895087 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ed30e3-6b87-4249-9716-c7524070e155" path="/var/lib/kubelet/pods/45ed30e3-6b87-4249-9716-c7524070e155/volumes" Nov 02 22:25:54.046147 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:54.046107 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f"] Nov 02 22:25:54.093065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:54.093032 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd3d5118-mns4f"] Nov 02 22:25:54.998235 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:54.998201 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj"] Nov 02 22:25:54.998994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:54.998975 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c66635b2-e0f2-4c30-8652-7fe13d667d27" containerName="revoke-keys" Nov 02 22:25:54.999101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:54.998996 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66635b2-e0f2-4c30-8652-7fe13d667d27" containerName="revoke-keys" Nov 02 22:25:54.999140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:54.999105 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66635b2-e0f2-4c30-8652-7fe13d667d27" containerName="revoke-keys" Nov 02 22:25:55.003017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.002995 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.102354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102313 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.102851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102381 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.102851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102553 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.102851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102680 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4ec82f-1f47-449f-8c45-244460554d96-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.102851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102733 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4ec82f-1f47-449f-8c45-244460554d96-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.102851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102789 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.103097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102886 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.103097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102933 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.103097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.102964 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsqs\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-kube-api-access-jnsqs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.106627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.106596 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj"] Nov 02 22:25:55.170753 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.170719 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv"] Nov 02 22:25:55.175118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.175096 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.203912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.203875 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.203952 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.203988 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204006 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsqs\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-kube-api-access-jnsqs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204060 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204101 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204139 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.204410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204192 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.204410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204392 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204454 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4ec82f-1f47-449f-8c45-244460554d96-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204490 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4ec82f-1f47-449f-8c45-244460554d96-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204510 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204513 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfwz\" (UniqueName: \"kubernetes.io/projected/1334fb99-90bf-4323-9bcf-409de877994c-kube-api-access-hxfwz\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.204614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204584 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204400 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.204922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.204659 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.205182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.205163 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4ec82f-1f47-449f-8c45-244460554d96-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.206821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.206803 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.207002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.206978 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4ec82f-1f47-449f-8c45-244460554d96-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.226619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.226548 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:25:55.226802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.226685 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:55.231938 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.231894 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv"] Nov 02 22:25:55.262954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.262912 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.263216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.263193 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsqs\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-kube-api-access-jnsqs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.305406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.305362 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.305652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.305431 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.305652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.305516 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfwz\" (UniqueName: \"kubernetes.io/projected/1334fb99-90bf-4323-9bcf-409de877994c-kube-api-access-hxfwz\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.306156 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.306135 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.308357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.308332 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.318313 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.318282 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:55.363529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.363496 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfwz\" (UniqueName: \"kubernetes.io/projected/1334fb99-90bf-4323-9bcf-409de877994c-kube-api-access-hxfwz\") pod \"maas-api-revoke-sub-0f60d85a-d4fjv\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.489886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.489849 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:55.588688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.588652 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t"] Nov 02 22:25:55.595672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.595639 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.653953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.652935 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t"] Nov 02 22:25:55.711382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711347 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.711382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711399 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.711706 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711428 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.711706 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711604 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlbn\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-kube-api-access-wqlbn\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.711706 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711649 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.711865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711782 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.711865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711823 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.712011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.711992 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.712072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.712058 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.753384 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.753347 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj"] Nov 02 22:25:55.755152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.755118 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:55.755254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.755188 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:55.755254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.755223 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:55.813193 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813161 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813193 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813212 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813248 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813278 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813300 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813494 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlbn\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-kube-api-access-wqlbn\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813590 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813626 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813682 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813716 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.813986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.813684 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.814090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.814024 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.814262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.814243 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.814510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.814266 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.816066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.816049 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.816455 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.816435 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.868005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.867964 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv"] Nov 02 22:25:55.872004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.871922 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlbn\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-kube-api-access-wqlbn\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.872004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.871929 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:55.897461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.897420 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2fbc73-440a-4209-8eec-a256c1310417" path="/var/lib/kubelet/pods/1d2fbc73-440a-4209-8eec-a256c1310417/volumes" Nov 02 22:25:55.916006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:55.915771 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:56.119094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.119049 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" event={"ID":"6f4ec82f-1f47-449f-8c45-244460554d96","Type":"ContainerStarted","Data":"e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d"} Nov 02 22:25:56.119711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.119683 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" event={"ID":"6f4ec82f-1f47-449f-8c45-244460554d96","Type":"ContainerStarted","Data":"7eb573e0d9bdd18c785361ebc566e9d7030ce3597a47d64da6921fc848afa9a4"} Nov 02 22:25:56.122142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.122110 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" event={"ID":"1334fb99-90bf-4323-9bcf-409de877994c","Type":"ContainerStarted","Data":"8774bfe84a2c0cca26d8d83d0931654d490c210e0d28f76873d9f878ab123d6a"} Nov 02 22:25:56.277119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.277055 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podStartSLOduration=2.277035965 podStartE2EDuration="2.277035965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:56.273495209 +0000 UTC m=+1192.538488429" watchObservedRunningTime="2026-11-02 22:25:56.277035965 +0000 UTC m=+1192.542029185" Nov 02 22:25:56.318885 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.318838 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:25:56.321325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.321227 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 22:25:56.321325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.321293 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 22:25:56.453428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.453390 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t"] Nov 02 22:25:56.455151 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:56.455114 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ce603b_0b3f_49f8_b801_3ba1d6c952e4.slice/crio-e2e113aa6e47a55f18fd14a9323dcf5f7922e5f0d18c71abe327f22e47ad2f95 WatchSource:0}: Error finding container e2e113aa6e47a55f18fd14a9323dcf5f7922e5f0d18c71abe327f22e47ad2f95: Status 404 returned error can't find the container with id e2e113aa6e47a55f18fd14a9323dcf5f7922e5f0d18c71abe327f22e47ad2f95 Nov 02 22:25:56.458030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.457994 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:56.458121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.458069 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:56.458121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.458102 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:56.495000 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:25:56.494965 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:25:56.611242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.611162 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw"] Nov 02 22:25:56.615782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.615751 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.646626 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.646702 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.646729 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.646789 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe162e44-0b4c-42a3-aacd-2abc635024b8-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.646969 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.647013 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnc4\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-kube-api-access-nvnc4\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.647047 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.647071 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.652869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.647162 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.688652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.688616 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw"] Nov 02 22:25:56.749657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749604 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749662 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnc4\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-kube-api-access-nvnc4\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749686 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749702 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749759 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749834 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749871 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749887 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.749944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.749924 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe162e44-0b4c-42a3-aacd-2abc635024b8-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.750900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.750731 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe162e44-0b4c-42a3-aacd-2abc635024b8-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.751028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.750997 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.751222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.751173 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.751294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.751231 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.751492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.751467 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.755063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.753876 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.755409 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.755387 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.808819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.808787 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.809089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.809063 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnc4\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-kube-api-access-nvnc4\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:56.958559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:56.958407 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:57.175789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.175175 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" event={"ID":"1334fb99-90bf-4323-9bcf-409de877994c","Type":"ContainerStarted","Data":"cd18747f4ebb39eb2eb2e7425d2ecc2f77349aff1a5fcf436e05c97ebab9337f"} Nov 02 22:25:57.180797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.180762 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" event={"ID":"40ce603b-0b3f-49f8-b801-3ba1d6c952e4","Type":"ContainerStarted","Data":"0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1"} Nov 02 22:25:57.181003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.180989 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" event={"ID":"40ce603b-0b3f-49f8-b801-3ba1d6c952e4","Type":"ContainerStarted","Data":"e2e113aa6e47a55f18fd14a9323dcf5f7922e5f0d18c71abe327f22e47ad2f95"} Nov 02 22:25:57.293842 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.293788 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" podStartSLOduration=2.293771407 podStartE2EDuration="2.293771407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:57.292038453 +0000 UTC m=+1193.557031672" watchObservedRunningTime="2026-11-02 22:25:57.293771407 +0000 UTC m=+1193.558764626" Nov 02 22:25:57.318714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.318680 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 22:25:57.318890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.318735 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 22:25:57.400440 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.400406 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw"] Nov 02 22:25:57.404711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.404673 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:57.404856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.404760 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:57.404856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.404808 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:25:57.471164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.447471 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podStartSLOduration=2.447444203 podStartE2EDuration="2.447444203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:57.40699562 +0000 UTC m=+1193.671988839" watchObservedRunningTime="2026-11-02 22:25:57.447444203 +0000 UTC m=+1193.712437423" Nov 02 22:25:57.848594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.847864 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g"] Nov 02 22:25:57.876593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.874880 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:57.890936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.890898 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:57.890936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.890964 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncn64\" (UniqueName: \"kubernetes.io/projected/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-kube-api-access-ncn64\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:57.890936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.891086 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:57.899988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.899957 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g"] Nov 02 22:25:57.917314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.917283 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:25:57.918056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.918028 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 22:25:57.918215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.918196 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 22:25:57.995939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.995900 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncn64\" (UniqueName: \"kubernetes.io/projected/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-kube-api-access-ncn64\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:57.996122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.996071 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:57.996350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.996325 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:57.997176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.997145 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:58.001820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:57.999415 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:58.084102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.084070 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncn64\" (UniqueName: \"kubernetes.io/projected/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-kube-api-access-ncn64\") pod \"maas-api-revoke-sub-11c1d5f1-88s8g\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:58.192623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.192516 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" event={"ID":"fe162e44-0b4c-42a3-aacd-2abc635024b8","Type":"ContainerStarted","Data":"8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680"} Nov 02 22:25:58.192623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.192612 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" event={"ID":"fe162e44-0b4c-42a3-aacd-2abc635024b8","Type":"ContainerStarted","Data":"1418da518a3f538bf605cf94054a6f901211dd024f91082d61cb4d13f5ff5967"} Nov 02 22:25:58.195526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.195493 5109 generic.go:356] "Generic (PLEG): container finished" podID="1334fb99-90bf-4323-9bcf-409de877994c" containerID="cd18747f4ebb39eb2eb2e7425d2ecc2f77349aff1a5fcf436e05c97ebab9337f" exitCode=0 Nov 02 22:25:58.195673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.195652 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" event={"ID":"1334fb99-90bf-4323-9bcf-409de877994c","Type":"ContainerDied","Data":"cd18747f4ebb39eb2eb2e7425d2ecc2f77349aff1a5fcf436e05c97ebab9337f"} Nov 02 22:25:58.203467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.203439 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:25:58.319432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.318972 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 22:25:58.319432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.319037 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 22:25:58.321390 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.321331 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podStartSLOduration=2.321311457 podStartE2EDuration="2.321311457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:58.315662992 +0000 UTC m=+1194.580656211" watchObservedRunningTime="2026-11-02 22:25:58.321311457 +0000 UTC m=+1194.586304678" Nov 02 22:25:58.917644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.917251 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 22:25:58.917644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.917301 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 22:25:58.974734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.974637 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:25:58.974916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.974880 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:25:58.975013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:58.974951 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:25:59.013937 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:25:59.012171 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec2f927_b2df_43fb_bac4_a30aeb20a3fa.slice/crio-a2460442438f347b0697bcfc9b39785765a297854aedc2d939f897d51126705f WatchSource:0}: Error finding container a2460442438f347b0697bcfc9b39785765a297854aedc2d939f897d51126705f: Status 404 returned error can't find the container with id a2460442438f347b0697bcfc9b39785765a297854aedc2d939f897d51126705f Nov 02 22:25:59.013937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.013681 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g"] Nov 02 22:25:59.217146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.216999 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" event={"ID":"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa","Type":"ContainerStarted","Data":"a2460442438f347b0697bcfc9b39785765a297854aedc2d939f897d51126705f"} Nov 02 22:25:59.321146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.320958 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 22:25:59.321146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.321022 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 22:25:59.918112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.917676 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 22:25:59.918112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.917732 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 22:25:59.932932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.932889 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:25:59.966049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.966014 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:25:59.966526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:25:59.966496 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:26:00.020499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.020463 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1334fb99-90bf-4323-9bcf-409de877994c-kube-api-access-hxfwz\" (UniqueName: \"kubernetes.io/projected/1334fb99-90bf-4323-9bcf-409de877994c-kube-api-access-hxfwz\") pod \"1334fb99-90bf-4323-9bcf-409de877994c\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " Nov 02 22:26:00.020733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.020642 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-xks-ca\") pod \"1334fb99-90bf-4323-9bcf-409de877994c\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " Nov 02 22:26:00.021089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.020808 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-openshift-ca\") pod \"1334fb99-90bf-4323-9bcf-409de877994c\" (UID: \"1334fb99-90bf-4323-9bcf-409de877994c\") " Nov 02 22:26:00.021220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.021173 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-openshift-ca" pod "1334fb99-90bf-4323-9bcf-409de877994c" (UID: "1334fb99-90bf-4323-9bcf-409de877994c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:00.021517 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.021502 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:00.023291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.023219 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-xks-ca" pod "1334fb99-90bf-4323-9bcf-409de877994c" (UID: "1334fb99-90bf-4323-9bcf-409de877994c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:00.023291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.023258 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1334fb99-90bf-4323-9bcf-409de877994c-kube-api-access-hxfwz" pod "1334fb99-90bf-4323-9bcf-409de877994c" (UID: "1334fb99-90bf-4323-9bcf-409de877994c"). InnerVolumeSpecName "kube-api-access-hxfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:00.123314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.123216 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxfwz\" (UniqueName: \"kubernetes.io/projected/1334fb99-90bf-4323-9bcf-409de877994c-kube-api-access-hxfwz\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:00.123314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.123254 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334fb99-90bf-4323-9bcf-409de877994c-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:00.340602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.336760 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 22:26:00.340602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.336833 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 22:26:00.522783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.522741 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" event={"ID":"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa","Type":"ContainerStarted","Data":"13a624a620b51479f6e162399ad0d423b4a0c6adc7ed95ac1df7eb55931a1cc6"} Nov 02 22:26:00.574250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.571258 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" Nov 02 22:26:00.575682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.574891 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv" event={"ID":"1334fb99-90bf-4323-9bcf-409de877994c","Type":"ContainerDied","Data":"8774bfe84a2c0cca26d8d83d0931654d490c210e0d28f76873d9f878ab123d6a"} Nov 02 22:26:00.575682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.574955 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8774bfe84a2c0cca26d8d83d0931654d490c210e0d28f76873d9f878ab123d6a" Nov 02 22:26:00.699595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.698879 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" podStartSLOduration=3.698858048 podStartE2EDuration="3.698858048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:00.698295049 +0000 UTC m=+1196.963288269" watchObservedRunningTime="2026-11-02 22:26:00.698858048 +0000 UTC m=+1196.963851268" Nov 02 22:26:00.916649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.916451 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 22:26:00.916649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.916519 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 22:26:00.969598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.969031 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:26:00.969598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:00.969100 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:26:01.318902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.318868 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 22:26:01.319467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.318921 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 22:26:01.607393 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.607300 5109 generic.go:356] "Generic (PLEG): container finished" podID="3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" containerID="13a624a620b51479f6e162399ad0d423b4a0c6adc7ed95ac1df7eb55931a1cc6" exitCode=0 Nov 02 22:26:01.607788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.607485 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" event={"ID":"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa","Type":"ContainerDied","Data":"13a624a620b51479f6e162399ad0d423b4a0c6adc7ed95ac1df7eb55931a1cc6"} Nov 02 22:26:01.916630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.916519 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 22:26:01.916887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.916854 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 22:26:01.973612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.965817 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:26:01.973612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:01.965883 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:26:02.319592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.319532 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 22:26:02.319804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.319612 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 22:26:02.916623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.916504 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 22:26:02.916623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.916561 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 22:26:02.961054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.961005 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:26:02.961267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.961074 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:26:02.980133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.980099 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:26:02.983804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.983772 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-kube-api-access-ncn64\" (UniqueName: \"kubernetes.io/projected/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-kube-api-access-ncn64\") pod \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " Nov 02 22:26:02.984048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.983933 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-xks-ca\") pod \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " Nov 02 22:26:02.984127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.984060 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-openshift-ca\") pod \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\" (UID: \"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa\") " Nov 02 22:26:02.985302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.985271 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-openshift-ca" pod "3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" (UID: "3ec2f927-b2df-43fb-bac4-a30aeb20a3fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:02.986468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.986436 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-kube-api-access-ncn64" pod "3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" (UID: "3ec2f927-b2df-43fb-bac4-a30aeb20a3fa"). InnerVolumeSpecName "kube-api-access-ncn64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:02.987234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:02.987193 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-xks-ca" pod "3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" (UID: "3ec2f927-b2df-43fb-bac4-a30aeb20a3fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:03.086189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.086151 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:03.086189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.086182 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:03.086189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.086194 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncn64\" (UniqueName: \"kubernetes.io/projected/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa-kube-api-access-ncn64\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:03.642595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.642368 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" event={"ID":"3ec2f927-b2df-43fb-bac4-a30aeb20a3fa","Type":"ContainerDied","Data":"a2460442438f347b0697bcfc9b39785765a297854aedc2d939f897d51126705f"} Nov 02 22:26:03.642595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.642475 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2460442438f347b0697bcfc9b39785765a297854aedc2d939f897d51126705f" Nov 02 22:26:03.642595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.642551 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g" Nov 02 22:26:03.795683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.795650 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:26:03.796171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.796139 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:26:03.796912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.796895 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:26:03.917054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.916948 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 22:26:03.917054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.917018 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 22:26:03.959076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.959033 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:26:03.959290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:03.959101 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:26:04.120029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.119385 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs"] Nov 02 22:26:04.120854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.120476 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1334fb99-90bf-4323-9bcf-409de877994c" containerName="revoke-keys" Nov 02 22:26:04.120854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.120535 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1334fb99-90bf-4323-9bcf-409de877994c" containerName="revoke-keys" Nov 02 22:26:04.120854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.120610 5109 cpu_manager.go:414] "containerMap: removing container" podUID="3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" containerName="revoke-keys" Nov 02 22:26:04.120854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.120619 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" containerName="revoke-keys" Nov 02 22:26:04.120854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.120847 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1334fb99-90bf-4323-9bcf-409de877994c" containerName="revoke-keys" Nov 02 22:26:04.121178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.120866 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" containerName="revoke-keys" Nov 02 22:26:04.128878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.128850 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.177335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.177186 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs"] Nov 02 22:26:04.188597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.180410 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:04.188597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.180532 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:26:04.207075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.206184 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthwv\" (UniqueName: \"kubernetes.io/projected/f7f26f78-6284-4545-93bb-8b2233a8119a-kube-api-access-zthwv\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.207075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.206285 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.207075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.206332 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.307196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.307154 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthwv\" (UniqueName: \"kubernetes.io/projected/f7f26f78-6284-4545-93bb-8b2233a8119a-kube-api-access-zthwv\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.307378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.307248 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.307378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.307291 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.308866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.308838 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.310400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.310376 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.368027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.367996 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthwv\" (UniqueName: \"kubernetes.io/projected/f7f26f78-6284-4545-93bb-8b2233a8119a-kube-api-access-zthwv\") pod \"maas-api-revoke-sub-80f32035-lmmhs\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.531135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.531087 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:04.921734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.921649 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:26:04.922237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.922217 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:26:04.922971 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.922949 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:26:04.959758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.959718 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:26:04.959910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.959790 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:26:04.979269 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:04.979239 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f26f78_6284_4545_93bb_8b2233a8119a.slice/crio-a9720b091109e56cf66ac5c86f7ae48be02b0d9660681d421c7a09ca6b9f3564 WatchSource:0}: Error finding container a9720b091109e56cf66ac5c86f7ae48be02b0d9660681d421c7a09ca6b9f3564: Status 404 returned error can't find the container with id a9720b091109e56cf66ac5c86f7ae48be02b0d9660681d421c7a09ca6b9f3564 Nov 02 22:26:04.979407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:04.979380 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs"] Nov 02 22:26:05.671681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:05.671635 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" event={"ID":"f7f26f78-6284-4545-93bb-8b2233a8119a","Type":"ContainerStarted","Data":"b4a87ed917f8bd0af2d15e5a70bab389172535560eae1416e496bbb3a3bf3678"} Nov 02 22:26:05.671863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:05.671714 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" event={"ID":"f7f26f78-6284-4545-93bb-8b2233a8119a","Type":"ContainerStarted","Data":"a9720b091109e56cf66ac5c86f7ae48be02b0d9660681d421c7a09ca6b9f3564"} Nov 02 22:26:05.784751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:05.784693 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" podStartSLOduration=1.784678131 podStartE2EDuration="1.784678131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:05.781656877 +0000 UTC m=+1202.046650109" watchObservedRunningTime="2026-11-02 22:26:05.784678131 +0000 UTC m=+1202.049671349" Nov 02 22:26:05.963600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:05.963509 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:26:05.964097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:05.964081 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:26:05.964874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:05.964858 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:26:06.681348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:06.681312 5109 generic.go:356] "Generic (PLEG): container finished" podID="f7f26f78-6284-4545-93bb-8b2233a8119a" containerID="b4a87ed917f8bd0af2d15e5a70bab389172535560eae1416e496bbb3a3bf3678" exitCode=0 Nov 02 22:26:06.681553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:06.681393 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" event={"ID":"f7f26f78-6284-4545-93bb-8b2233a8119a","Type":"ContainerDied","Data":"b4a87ed917f8bd0af2d15e5a70bab389172535560eae1416e496bbb3a3bf3678"} Nov 02 22:26:07.311284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.311241 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c"] Nov 02 22:26:07.316170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.316140 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.369093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.369060 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.369299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.369126 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvrjt\" (UniqueName: \"kubernetes.io/projected/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-kube-api-access-bvrjt\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.369370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.369302 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.372862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.372798 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c"] Nov 02 22:26:07.470994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.470954 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.471184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.471166 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.471228 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.471195 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrjt\" (UniqueName: \"kubernetes.io/projected/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-kube-api-access-bvrjt\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.471901 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.471878 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.473492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.473467 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.530125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.530085 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrjt\" (UniqueName: \"kubernetes.io/projected/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-kube-api-access-bvrjt\") pod \"maas-api-revoke-sub-b42cbaaf-k4w2c\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.630052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.629964 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:07.884993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.884918 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:07.979121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.979086 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-xks-ca\") pod \"f7f26f78-6284-4545-93bb-8b2233a8119a\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " Nov 02 22:26:07.979409 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.979388 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-openshift-ca\") pod \"f7f26f78-6284-4545-93bb-8b2233a8119a\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " Nov 02 22:26:07.979479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.979463 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f26f78-6284-4545-93bb-8b2233a8119a-kube-api-access-zthwv\" (UniqueName: \"kubernetes.io/projected/f7f26f78-6284-4545-93bb-8b2233a8119a-kube-api-access-zthwv\") pod \"f7f26f78-6284-4545-93bb-8b2233a8119a\" (UID: \"f7f26f78-6284-4545-93bb-8b2233a8119a\") " Nov 02 22:26:07.979791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.979763 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-openshift-ca" pod "f7f26f78-6284-4545-93bb-8b2233a8119a" (UID: "f7f26f78-6284-4545-93bb-8b2233a8119a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:07.980248 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.980146 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:07.981655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.981633 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-xks-ca" pod "f7f26f78-6284-4545-93bb-8b2233a8119a" (UID: "f7f26f78-6284-4545-93bb-8b2233a8119a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:07.981787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:07.981767 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f26f78-6284-4545-93bb-8b2233a8119a-kube-api-access-zthwv" pod "f7f26f78-6284-4545-93bb-8b2233a8119a" (UID: "f7f26f78-6284-4545-93bb-8b2233a8119a"). InnerVolumeSpecName "kube-api-access-zthwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:08.009184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.009133 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c"] Nov 02 22:26:08.015128 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:08.015096 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfe17d0_ba17_4dd4_9a79_8d573b292b53.slice/crio-f84a7b86237f82f68800e49501279381cb638b3420adb897c7a881c1b5799b2d WatchSource:0}: Error finding container f84a7b86237f82f68800e49501279381cb638b3420adb897c7a881c1b5799b2d: Status 404 returned error can't find the container with id f84a7b86237f82f68800e49501279381cb638b3420adb897c7a881c1b5799b2d Nov 02 22:26:08.081647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.081602 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f26f78-6284-4545-93bb-8b2233a8119a-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:08.081647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.081640 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zthwv\" (UniqueName: \"kubernetes.io/projected/f7f26f78-6284-4545-93bb-8b2233a8119a-kube-api-access-zthwv\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:08.708899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.708872 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" Nov 02 22:26:08.709367 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.708867 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs" event={"ID":"f7f26f78-6284-4545-93bb-8b2233a8119a","Type":"ContainerDied","Data":"a9720b091109e56cf66ac5c86f7ae48be02b0d9660681d421c7a09ca6b9f3564"} Nov 02 22:26:08.709367 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.709005 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9720b091109e56cf66ac5c86f7ae48be02b0d9660681d421c7a09ca6b9f3564" Nov 02 22:26:08.710946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.710915 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" event={"ID":"4cfe17d0-ba17-4dd4-9a79-8d573b292b53","Type":"ContainerStarted","Data":"001b506a1077449e2bb21b9c45c4b9eb2fc22a44f060c49a9d0f6e60f3482643"} Nov 02 22:26:08.711094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.710962 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" event={"ID":"4cfe17d0-ba17-4dd4-9a79-8d573b292b53","Type":"ContainerStarted","Data":"f84a7b86237f82f68800e49501279381cb638b3420adb897c7a881c1b5799b2d"} Nov 02 22:26:08.824804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:08.824748 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" podStartSLOduration=1.8247318959999999 podStartE2EDuration="1.824731896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:08.820237155 +0000 UTC m=+1205.085230374" watchObservedRunningTime="2026-11-02 22:26:08.824731896 +0000 UTC m=+1205.089725153" Nov 02 22:26:09.724236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:09.724196 5109 generic.go:356] "Generic (PLEG): container finished" podID="4cfe17d0-ba17-4dd4-9a79-8d573b292b53" containerID="001b506a1077449e2bb21b9c45c4b9eb2fc22a44f060c49a9d0f6e60f3482643" exitCode=0 Nov 02 22:26:09.724779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:09.724295 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" event={"ID":"4cfe17d0-ba17-4dd4-9a79-8d573b292b53","Type":"ContainerDied","Data":"001b506a1077449e2bb21b9c45c4b9eb2fc22a44f060c49a9d0f6e60f3482643"} Nov 02 22:26:11.150990 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.150957 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:11.205909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.205862 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-xks-ca\") pod \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " Nov 02 22:26:11.206110 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.205946 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-kube-api-access-bvrjt\" (UniqueName: \"kubernetes.io/projected/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-kube-api-access-bvrjt\") pod \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " Nov 02 22:26:11.206178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.206164 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-openshift-ca\") pod \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\" (UID: \"4cfe17d0-ba17-4dd4-9a79-8d573b292b53\") " Nov 02 22:26:11.206574 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.206542 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-openshift-ca" pod "4cfe17d0-ba17-4dd4-9a79-8d573b292b53" (UID: "4cfe17d0-ba17-4dd4-9a79-8d573b292b53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:11.206806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.206786 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:11.208922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.208896 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-kube-api-access-bvrjt" pod "4cfe17d0-ba17-4dd4-9a79-8d573b292b53" (UID: "4cfe17d0-ba17-4dd4-9a79-8d573b292b53"). InnerVolumeSpecName "kube-api-access-bvrjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:11.209360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.209333 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-xks-ca" pod "4cfe17d0-ba17-4dd4-9a79-8d573b292b53" (UID: "4cfe17d0-ba17-4dd4-9a79-8d573b292b53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:11.319064 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.318985 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:11.319064 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.319030 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvrjt\" (UniqueName: \"kubernetes.io/projected/4cfe17d0-ba17-4dd4-9a79-8d573b292b53-kube-api-access-bvrjt\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:11.766600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.765806 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" event={"ID":"4cfe17d0-ba17-4dd4-9a79-8d573b292b53","Type":"ContainerDied","Data":"f84a7b86237f82f68800e49501279381cb638b3420adb897c7a881c1b5799b2d"} Nov 02 22:26:11.766600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.765875 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f84a7b86237f82f68800e49501279381cb638b3420adb897c7a881c1b5799b2d" Nov 02 22:26:11.766600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:11.765951 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c" Nov 02 22:26:12.076670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:12.075609 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf"] Nov 02 22:26:12.125065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:12.125023 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc6b28e-2nmsf"] Nov 02 22:26:13.291786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:13.291722 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.533752 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:13.533697 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.587834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:13.587708 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.816595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:13.802673 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.920653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:13.888069 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.920973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:13.920925 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.969592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:13.963259 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3646f5b-eccd-4afe-8074-2889bb75a558" path="/var/lib/kubelet/pods/e3646f5b-eccd-4afe-8074-2889bb75a558/volumes" Nov 02 22:26:16.050613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:16.050560 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj"] Nov 02 22:26:16.096978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:16.096939 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026aaab9-trrtj"] Nov 02 22:26:17.893906 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:17.893873 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bcda0b-bc5d-4131-b15c-16339702817a" path="/var/lib/kubelet/pods/10bcda0b-bc5d-4131-b15c-16339702817a/volumes" Nov 02 22:26:20.806727 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.806688 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq"] Nov 02 22:26:20.807480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.807459 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f7f26f78-6284-4545-93bb-8b2233a8119a" containerName="revoke-keys" Nov 02 22:26:20.807524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.807483 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f26f78-6284-4545-93bb-8b2233a8119a" containerName="revoke-keys" Nov 02 22:26:20.807558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.807535 5109 cpu_manager.go:414] "containerMap: removing container" podUID="4cfe17d0-ba17-4dd4-9a79-8d573b292b53" containerName="revoke-keys" Nov 02 22:26:20.807558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.807542 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfe17d0-ba17-4dd4-9a79-8d573b292b53" containerName="revoke-keys" Nov 02 22:26:20.807742 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.807728 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f26f78-6284-4545-93bb-8b2233a8119a" containerName="revoke-keys" Nov 02 22:26:20.807784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.807753 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfe17d0-ba17-4dd4-9a79-8d573b292b53" containerName="revoke-keys" Nov 02 22:26:20.812116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.812094 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:20.868005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.867963 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq"] Nov 02 22:26:20.907301 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.907263 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:20.907472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.907322 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:26:20.910107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.910077 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:20.910292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.910278 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:20.910350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:20.910327 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf44n\" (UniqueName: \"kubernetes.io/projected/8bb0b162-af08-45b2-b477-a77c3f3e89c8-kube-api-access-kf44n\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.012068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.012028 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.012260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.012191 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.012260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.012241 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf44n\" (UniqueName: \"kubernetes.io/projected/8bb0b162-af08-45b2-b477-a77c3f3e89c8-kube-api-access-kf44n\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.012910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.012825 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.014640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.014621 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.070777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.070695 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf44n\" (UniqueName: \"kubernetes.io/projected/8bb0b162-af08-45b2-b477-a77c3f3e89c8-kube-api-access-kf44n\") pod \"maas-api-revoke-sub-f83b946d-x6gmq\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.129780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.129743 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:21.375721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.375638 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m"] Nov 02 22:26:21.380438 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.380412 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.435963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.435925 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m"] Nov 02 22:26:21.518970 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.518933 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.519149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.518980 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.519149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.519122 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg42m\" (UniqueName: \"kubernetes.io/projected/35c9fcdd-da7b-472f-a922-d18360ec9bd6-kube-api-access-gg42m\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.541017 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:21.540971 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb0b162_af08_45b2_b477_a77c3f3e89c8.slice/crio-db0e0e01ab560c3d5f61ecf0a8b3acaaf7a5fcc9991492996db8a15177bc4039 WatchSource:0}: Error finding container db0e0e01ab560c3d5f61ecf0a8b3acaaf7a5fcc9991492996db8a15177bc4039: Status 404 returned error can't find the container with id db0e0e01ab560c3d5f61ecf0a8b3acaaf7a5fcc9991492996db8a15177bc4039 Nov 02 22:26:21.541675 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.541650 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq"] Nov 02 22:26:21.620741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.620704 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.620741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.620749 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.621032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.620828 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg42m\" (UniqueName: \"kubernetes.io/projected/35c9fcdd-da7b-472f-a922-d18360ec9bd6-kube-api-access-gg42m\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.621418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.621394 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.623415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.623384 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.679257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.679180 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg42m\" (UniqueName: \"kubernetes.io/projected/35c9fcdd-da7b-472f-a922-d18360ec9bd6-kube-api-access-gg42m\") pod \"maas-api-revoke-sub-0e11d791-kf28m\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.695613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.695554 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:21.943288 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:21.943250 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" event={"ID":"8bb0b162-af08-45b2-b477-a77c3f3e89c8","Type":"ContainerStarted","Data":"db0e0e01ab560c3d5f61ecf0a8b3acaaf7a5fcc9991492996db8a15177bc4039"} Nov 02 22:26:22.072711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:22.072669 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m"] Nov 02 22:26:22.077383 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:22.077352 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c9fcdd_da7b_472f_a922_d18360ec9bd6.slice/crio-d4c47f03be2f8a1af1d7ef90d719862a4e33aba49550fc4acc80932c378c28a5 WatchSource:0}: Error finding container d4c47f03be2f8a1af1d7ef90d719862a4e33aba49550fc4acc80932c378c28a5: Status 404 returned error can't find the container with id d4c47f03be2f8a1af1d7ef90d719862a4e33aba49550fc4acc80932c378c28a5 Nov 02 22:26:22.953365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:22.953318 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" event={"ID":"35c9fcdd-da7b-472f-a922-d18360ec9bd6","Type":"ContainerStarted","Data":"08614d4c2c52acb29553e643e43821f05133822325ff7349e294ec6c1cb5a7a5"} Nov 02 22:26:22.953870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:22.953383 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" event={"ID":"35c9fcdd-da7b-472f-a922-d18360ec9bd6","Type":"ContainerStarted","Data":"d4c47f03be2f8a1af1d7ef90d719862a4e33aba49550fc4acc80932c378c28a5"} Nov 02 22:26:22.955256 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:22.955224 5109 generic.go:356] "Generic (PLEG): container finished" podID="8bb0b162-af08-45b2-b477-a77c3f3e89c8" containerID="04afe1535124d4d1ab8758fba89e4cbe4b856cfea7be2fb89ac3d7c4090be4ae" exitCode=0 Nov 02 22:26:22.955370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:22.955306 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" event={"ID":"8bb0b162-af08-45b2-b477-a77c3f3e89c8","Type":"ContainerDied","Data":"04afe1535124d4d1ab8758fba89e4cbe4b856cfea7be2fb89ac3d7c4090be4ae"} Nov 02 22:26:23.054325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:23.054285 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv"] Nov 02 22:26:23.058833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:23.058797 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2360b032-jjcwv"] Nov 02 22:26:23.097975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:23.097922 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" podStartSLOduration=2.097907166 podStartE2EDuration="2.097907166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:23.094613344 +0000 UTC m=+1219.359606554" watchObservedRunningTime="2026-11-02 22:26:23.097907166 +0000 UTC m=+1219.362900413" Nov 02 22:26:23.894750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:23.894718 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8914c825-df1b-498f-b342-3593a4d2f331" path="/var/lib/kubelet/pods/8914c825-df1b-498f-b342-3593a4d2f331/volumes" Nov 02 22:26:23.965409 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:23.965377 5109 generic.go:356] "Generic (PLEG): container finished" podID="35c9fcdd-da7b-472f-a922-d18360ec9bd6" containerID="08614d4c2c52acb29553e643e43821f05133822325ff7349e294ec6c1cb5a7a5" exitCode=0 Nov 02 22:26:23.965868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:23.965469 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" event={"ID":"35c9fcdd-da7b-472f-a922-d18360ec9bd6","Type":"ContainerDied","Data":"08614d4c2c52acb29553e643e43821f05133822325ff7349e294ec6c1cb5a7a5"} Nov 02 22:26:24.051388 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.051347 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx"] Nov 02 22:26:24.066594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.066549 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1484f6-gwbpx"] Nov 02 22:26:24.134106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.134081 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:24.261238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.261204 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-xks-ca\") pod \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " Nov 02 22:26:24.261419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.261371 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-openshift-ca\") pod \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " Nov 02 22:26:24.261532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.261514 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb0b162-af08-45b2-b477-a77c3f3e89c8-kube-api-access-kf44n\" (UniqueName: \"kubernetes.io/projected/8bb0b162-af08-45b2-b477-a77c3f3e89c8-kube-api-access-kf44n\") pod \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\" (UID: \"8bb0b162-af08-45b2-b477-a77c3f3e89c8\") " Nov 02 22:26:24.261746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.261720 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-openshift-ca" pod "8bb0b162-af08-45b2-b477-a77c3f3e89c8" (UID: "8bb0b162-af08-45b2-b477-a77c3f3e89c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:24.262380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.262353 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:24.263833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.263808 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-xks-ca" pod "8bb0b162-af08-45b2-b477-a77c3f3e89c8" (UID: "8bb0b162-af08-45b2-b477-a77c3f3e89c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:24.264734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.264710 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb0b162-af08-45b2-b477-a77c3f3e89c8-kube-api-access-kf44n" pod "8bb0b162-af08-45b2-b477-a77c3f3e89c8" (UID: "8bb0b162-af08-45b2-b477-a77c3f3e89c8"). InnerVolumeSpecName "kube-api-access-kf44n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:24.363373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.363332 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb0b162-af08-45b2-b477-a77c3f3e89c8-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:24.363373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.363368 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf44n\" (UniqueName: \"kubernetes.io/projected/8bb0b162-af08-45b2-b477-a77c3f3e89c8-kube-api-access-kf44n\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:24.908868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.908828 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75"] Nov 02 22:26:24.909728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.909705 5109 cpu_manager.go:414] "containerMap: removing container" podUID="8bb0b162-af08-45b2-b477-a77c3f3e89c8" containerName="revoke-keys" Nov 02 22:26:24.909806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.909731 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb0b162-af08-45b2-b477-a77c3f3e89c8" containerName="revoke-keys" Nov 02 22:26:24.909920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.909907 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb0b162-af08-45b2-b477-a77c3f3e89c8" containerName="revoke-keys" Nov 02 22:26:24.913652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.913626 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:24.955380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.955335 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d"] Nov 02 22:26:24.960066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.960041 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:24.976634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.976604 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" event={"ID":"8bb0b162-af08-45b2-b477-a77c3f3e89c8","Type":"ContainerDied","Data":"db0e0e01ab560c3d5f61ecf0a8b3acaaf7a5fcc9991492996db8a15177bc4039"} Nov 02 22:26:24.976996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.976647 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq" Nov 02 22:26:24.976996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:24.976682 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db0e0e01ab560c3d5f61ecf0a8b3acaaf7a5fcc9991492996db8a15177bc4039" Nov 02 22:26:25.011141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.010770 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75"] Nov 02 22:26:25.061259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.061222 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d"] Nov 02 22:26:25.073755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.073719 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fe2fec-540b-473f-8303-46752a067bdc-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.073942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.073766 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.073942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.073864 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-kube-api-access-6x8xq\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.073942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.073895 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.073942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.073931 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb518472-d243-4a01-aead-146570bff41f-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.074148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074115 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.074219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074201 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45v4x\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-kube-api-access-45v4x\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.074268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074238 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.074407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074391 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb518472-d243-4a01-aead-146570bff41f-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.074471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074442 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.074731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074535 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.074731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074692 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fe2fec-540b-473f-8303-46752a067bdc-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.074894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074848 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.074894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074873 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.075060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074909 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.075060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074959 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.075060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.074979 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.075215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.075086 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.146735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.146709 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:25.176881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.176785 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fe2fec-540b-473f-8303-46752a067bdc-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.176881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.176825 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.176881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.176855 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-kube-api-access-6x8xq\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.177165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.176886 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.177342 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177317 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.177592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177546 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb518472-d243-4a01-aead-146570bff41f-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.177683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177595 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fe2fec-540b-473f-8303-46752a067bdc-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.177683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177622 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.177791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177674 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.177791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177757 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45v4x\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-kube-api-access-45v4x\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.177892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177800 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.177892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177886 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb518472-d243-4a01-aead-146570bff41f-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.177994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177914 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.177994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.177935 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.178094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178002 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.178094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178060 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fe2fec-540b-473f-8303-46752a067bdc-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.178197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178126 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.178197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178159 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.178197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178160 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.178343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178221 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.178343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178254 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb518472-d243-4a01-aead-146570bff41f-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.178343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178260 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.178343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178285 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.178529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178377 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.178801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.178775 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.179075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.179052 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.179334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.179100 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.179691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.179668 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.180863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.180837 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.181214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.181193 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fe2fec-540b-473f-8303-46752a067bdc-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.181325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.181304 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.181723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.181703 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb518472-d243-4a01-aead-146570bff41f-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.236153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.236115 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.236153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.236148 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-kube-api-access-6x8xq\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.236387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.236185 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.236387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.236261 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45v4x\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-kube-api-access-45v4x\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.275728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.275690 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:25.278998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.278967 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-xks-ca\") pod \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " Nov 02 22:26:25.279165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.279026 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c9fcdd-da7b-472f-a922-d18360ec9bd6-kube-api-access-gg42m\" (UniqueName: \"kubernetes.io/projected/35c9fcdd-da7b-472f-a922-d18360ec9bd6-kube-api-access-gg42m\") pod \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " Nov 02 22:26:25.279348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.279332 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-openshift-ca\") pod \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\" (UID: \"35c9fcdd-da7b-472f-a922-d18360ec9bd6\") " Nov 02 22:26:25.279768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.279744 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-openshift-ca" pod "35c9fcdd-da7b-472f-a922-d18360ec9bd6" (UID: "35c9fcdd-da7b-472f-a922-d18360ec9bd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:25.280002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.279980 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.281443 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.281420 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-xks-ca" pod "35c9fcdd-da7b-472f-a922-d18360ec9bd6" (UID: "35c9fcdd-da7b-472f-a922-d18360ec9bd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:25.281788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.281766 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c9fcdd-da7b-472f-a922-d18360ec9bd6-kube-api-access-gg42m" pod "35c9fcdd-da7b-472f-a922-d18360ec9bd6" (UID: "35c9fcdd-da7b-472f-a922-d18360ec9bd6"). InnerVolumeSpecName "kube-api-access-gg42m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:25.382119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.382068 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c9fcdd-da7b-472f-a922-d18360ec9bd6-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.382119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.382118 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg42m\" (UniqueName: \"kubernetes.io/projected/35c9fcdd-da7b-472f-a922-d18360ec9bd6-kube-api-access-gg42m\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.535433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.535343 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:25.706791 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:25.706755 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb518472_d243_4a01_aead_146570bff41f.slice/crio-49e8fd79313fd2a49b7a7ddfe55bcf996bf5ddbe9a944d48702d74504ba9b981 WatchSource:0}: Error finding container 49e8fd79313fd2a49b7a7ddfe55bcf996bf5ddbe9a944d48702d74504ba9b981: Status 404 returned error can't find the container with id 49e8fd79313fd2a49b7a7ddfe55bcf996bf5ddbe9a944d48702d74504ba9b981 Nov 02 22:26:25.707405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.707370 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d"] Nov 02 22:26:25.709550 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.709508 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:25.709689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.709612 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:25.709689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.709659 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:25.897194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.897148 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832747a8-354a-4109-9fc5-b54c2d5808dd" path="/var/lib/kubelet/pods/832747a8-354a-4109-9fc5-b54c2d5808dd/volumes" Nov 02 22:26:25.925324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.922587 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75"] Nov 02 22:26:25.928932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.928893 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:25.929067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.928982 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:25.929067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.929029 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:25.994501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.992648 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" Nov 02 22:26:25.994501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.992647 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m" event={"ID":"35c9fcdd-da7b-472f-a922-d18360ec9bd6","Type":"ContainerDied","Data":"d4c47f03be2f8a1af1d7ef90d719862a4e33aba49550fc4acc80932c378c28a5"} Nov 02 22:26:25.994501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.992793 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4c47f03be2f8a1af1d7ef90d719862a4e33aba49550fc4acc80932c378c28a5" Nov 02 22:26:25.995912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.995879 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" event={"ID":"e9fe2fec-540b-473f-8303-46752a067bdc","Type":"ContainerStarted","Data":"664d75c6cb75047a03bd3307ddbbe8a4df305d9569edd72b842791218b56e9c8"} Nov 02 22:26:25.998404 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.998377 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" event={"ID":"fb518472-d243-4a01-aead-146570bff41f","Type":"ContainerStarted","Data":"969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3"} Nov 02 22:26:25.998525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:25.998433 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" event={"ID":"fb518472-d243-4a01-aead-146570bff41f","Type":"ContainerStarted","Data":"49e8fd79313fd2a49b7a7ddfe55bcf996bf5ddbe9a944d48702d74504ba9b981"} Nov 02 22:26:26.051219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.051122 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk"] Nov 02 22:26:26.058319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.058247 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b5ceffc-jhspk"] Nov 02 22:26:26.195391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.195311 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podStartSLOduration=2.195288034 podStartE2EDuration="2.195288034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:26.163798142 +0000 UTC m=+1222.428791361" watchObservedRunningTime="2026-11-02 22:26:26.195288034 +0000 UTC m=+1222.460281254" Nov 02 22:26:26.198686 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.198652 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z"] Nov 02 22:26:26.199489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.199470 5109 cpu_manager.go:414] "containerMap: removing container" podUID="35c9fcdd-da7b-472f-a922-d18360ec9bd6" containerName="revoke-keys" Nov 02 22:26:26.199489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.199490 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c9fcdd-da7b-472f-a922-d18360ec9bd6" containerName="revoke-keys" Nov 02 22:26:26.199676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.199660 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c9fcdd-da7b-472f-a922-d18360ec9bd6" containerName="revoke-keys" Nov 02 22:26:26.203703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.203682 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.280984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.280949 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:26.282394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.282365 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:26.282464 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.282426 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:26.290605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.285890 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z"] Nov 02 22:26:26.311721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.308635 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.311721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.308688 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.311721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.308865 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.311721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.308940 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgcr\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-kube-api-access-ddgcr\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.311721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.308975 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.313618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.312523 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.313618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.312671 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08b9afc-31fe-4677-82fd-0eb278017edf-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.313618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.312734 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.313618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.312830 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.414435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414398 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.414693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414502 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.414693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414622 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08b9afc-31fe-4677-82fd-0eb278017edf-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.414693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414679 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.414874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414761 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.414922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414872 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.414922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414896 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.415032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.414956 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.415032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.415004 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.415144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.415088 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.415144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.415129 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.415239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.415163 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgcr\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-kube-api-access-ddgcr\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.415483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.415452 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08b9afc-31fe-4677-82fd-0eb278017edf-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.415483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.415452 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.417628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.417604 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.417857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.417833 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.474982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.474556 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.475793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.475309 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgcr\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-kube-api-access-ddgcr\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:26.503213 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:26:26.503169 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:26:26.521988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:26.521951 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:27.008995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.008947 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" event={"ID":"e9fe2fec-540b-473f-8303-46752a067bdc","Type":"ContainerStarted","Data":"6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b"} Nov 02 22:26:27.078255 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:27.075707 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08b9afc_31fe_4677_82fd_0eb278017edf.slice/crio-3ab9ce912c3a699b48920895cda45fa039cb9974a3ea31412affd4dba85ddd5d WatchSource:0}: Error finding container 3ab9ce912c3a699b48920895cda45fa039cb9974a3ea31412affd4dba85ddd5d: Status 404 returned error can't find the container with id 3ab9ce912c3a699b48920895cda45fa039cb9974a3ea31412affd4dba85ddd5d Nov 02 22:26:27.086374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.086336 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z"] Nov 02 22:26:27.089788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.089743 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:27.090035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.090011 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:27.090228 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.090206 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:26:27.182715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.182651 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podStartSLOduration=3.182632864 podStartE2EDuration="3.182632864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:27.174082812 +0000 UTC m=+1223.439076066" watchObservedRunningTime="2026-11-02 22:26:27.182632864 +0000 UTC m=+1223.447626083" Nov 02 22:26:27.282653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.277102 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:27.282653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.277188 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:27.545973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.544255 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:27.572594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.546743 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:26:27.572594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.547009 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:26:27.901287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:27.901212 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45438ae-ff43-4c35-83d2-4f02237d6aac" path="/var/lib/kubelet/pods/e45438ae-ff43-4c35-83d2-4f02237d6aac/volumes" Nov 02 22:26:28.020929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.020887 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" event={"ID":"c08b9afc-31fe-4677-82fd-0eb278017edf","Type":"ContainerStarted","Data":"b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e"} Nov 02 22:26:28.021438 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.020955 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" event={"ID":"c08b9afc-31fe-4677-82fd-0eb278017edf","Type":"ContainerStarted","Data":"3ab9ce912c3a699b48920895cda45fa039cb9974a3ea31412affd4dba85ddd5d"} Nov 02 22:26:28.291102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.291051 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:28.291396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.291368 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:28.522230 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.522188 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:28.535596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.531712 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:26:28.535596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.531785 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 22:26:28.540594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.537045 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:26:28.540594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:28.537115 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:26:29.082596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.079857 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podStartSLOduration=3.079833617 podStartE2EDuration="3.079833617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:28.180035966 +0000 UTC m=+1224.445029186" watchObservedRunningTime="2026-11-02 22:26:29.079833617 +0000 UTC m=+1225.344826838" Nov 02 22:26:29.102278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.099673 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc"] Nov 02 22:26:29.108373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.107795 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df893d0b-s9qzc"] Nov 02 22:26:29.277601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.277493 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:29.277601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.277583 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:29.522489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.522450 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:26:29.522718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.522503 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 22:26:29.536203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.536054 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:26:29.536203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.536126 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:26:29.604691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.601900 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb"] Nov 02 22:26:29.610225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.610196 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.681593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.676661 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:29.681593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.676801 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:26:29.703626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.703046 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.703626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.703162 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh57s\" (UniqueName: \"kubernetes.io/projected/d219b9b9-b686-4d46-b944-87bb6f9e022b-kube-api-access-sh57s\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.703626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.703343 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.720001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.718985 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb"] Nov 02 22:26:29.811827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.811614 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.811827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.811735 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.812061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.811860 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh57s\" (UniqueName: \"kubernetes.io/projected/d219b9b9-b686-4d46-b944-87bb6f9e022b-kube-api-access-sh57s\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.816124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.814361 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.818018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.817986 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.915644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.915551 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh57s\" (UniqueName: \"kubernetes.io/projected/d219b9b9-b686-4d46-b944-87bb6f9e022b-kube-api-access-sh57s\") pod \"maas-api-revoke-sub-d3ac0ac9-r69lb\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:29.959593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:29.954821 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc9a5a3-6d45-4675-a695-6304f3cc7dac" path="/var/lib/kubelet/pods/2cc9a5a3-6d45-4675-a695-6304f3cc7dac/volumes" Nov 02 22:26:30.055594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:30.053827 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:30.309596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:30.309006 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:30.309596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:30.309067 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:30.522943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:30.522907 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:26:30.523109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:30.522967 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 22:26:30.560606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:30.560037 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:26:30.560606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:30.560101 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:26:31.025594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.025101 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb"] Nov 02 22:26:31.220027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.219989 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" event={"ID":"d219b9b9-b686-4d46-b944-87bb6f9e022b","Type":"ContainerStarted","Data":"97720233d2e231d5a4b3a474d1eaa6a8a36623b0b9dcbdd466a62496b387835f"} Nov 02 22:26:31.297596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.293455 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:31.297596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.293537 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:31.523122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.523079 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:26:31.523620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.523157 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 22:26:31.544595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.544076 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:26:31.544595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:31.544147 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:26:32.294041 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.293991 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:32.294348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.294305 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:32.297925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.297891 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" event={"ID":"d219b9b9-b686-4d46-b944-87bb6f9e022b","Type":"ContainerStarted","Data":"61238c1bb9b4660f514569a77f8f60dae061434a1366984e4563e0d3c2c854ad"} Nov 02 22:26:32.419365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.419298 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" podStartSLOduration=3.419275966 podStartE2EDuration="3.419275966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:32.417312614 +0000 UTC m=+1228.682305844" watchObservedRunningTime="2026-11-02 22:26:32.419275966 +0000 UTC m=+1228.684269185" Nov 02 22:26:32.523253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.523214 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:26:32.523811 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.523292 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 22:26:32.541593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.539027 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:26:32.541593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.539096 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:26:32.571134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.571065 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp"] Nov 02 22:26:32.577076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.577045 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.683752 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.683720 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp"] Nov 02 22:26:32.702071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.702025 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.702223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.702146 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.702410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.702382 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpl9\" (UniqueName: \"kubernetes.io/projected/c07398f0-55be-42f6-a86e-39d2f96f77fa-kube-api-access-5kpl9\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.804472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.804437 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpl9\" (UniqueName: \"kubernetes.io/projected/c07398f0-55be-42f6-a86e-39d2f96f77fa-kube-api-access-5kpl9\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.804684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.804596 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.804684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.804668 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.805553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.805493 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.807535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.807512 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.872669 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.870885 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpl9\" (UniqueName: \"kubernetes.io/projected/c07398f0-55be-42f6-a86e-39d2f96f77fa-kube-api-access-5kpl9\") pod \"maas-api-revoke-sub-11a045f2-b7dzp\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:32.992063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:32.992010 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:33.276951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.276444 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:26:33.276951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.276505 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 02 22:26:33.308671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.308632 5109 generic.go:356] "Generic (PLEG): container finished" podID="d219b9b9-b686-4d46-b944-87bb6f9e022b" containerID="61238c1bb9b4660f514569a77f8f60dae061434a1366984e4563e0d3c2c854ad" exitCode=0 Nov 02 22:26:33.308863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.308739 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" event={"ID":"d219b9b9-b686-4d46-b944-87bb6f9e022b","Type":"ContainerDied","Data":"61238c1bb9b4660f514569a77f8f60dae061434a1366984e4563e0d3c2c854ad"} Nov 02 22:26:33.415647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.415617 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp"] Nov 02 22:26:33.416374 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:33.416339 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07398f0_55be_42f6_a86e_39d2f96f77fa.slice/crio-4c35f1ec9b45761d5eea3b219a91fbd7611efb7fbb286f3bd9c70b5e619645d7 WatchSource:0}: Error finding container 4c35f1ec9b45761d5eea3b219a91fbd7611efb7fbb286f3bd9c70b5e619645d7: Status 404 returned error can't find the container with id 4c35f1ec9b45761d5eea3b219a91fbd7611efb7fbb286f3bd9c70b5e619645d7 Nov 02 22:26:33.523139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.523050 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:26:33.523139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.523111 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 22:26:33.539591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.536445 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:26:33.539591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:33.536510 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:26:34.056027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.055985 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl"] Nov 02 22:26:34.063552 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.063508 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fee2d8-2m9jl"] Nov 02 22:26:34.282776 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.282743 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:34.283216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.283193 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:34.284219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.284198 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:26:34.318598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.318544 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" event={"ID":"c07398f0-55be-42f6-a86e-39d2f96f77fa","Type":"ContainerStarted","Data":"b31427295f0119854dbc58f1282807eae9229d21a279eb0f80c3a3113e312006"} Nov 02 22:26:34.318808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.318626 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" event={"ID":"c07398f0-55be-42f6-a86e-39d2f96f77fa","Type":"ContainerStarted","Data":"4c35f1ec9b45761d5eea3b219a91fbd7611efb7fbb286f3bd9c70b5e619645d7"} Nov 02 22:26:34.523104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.523065 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:26:34.523279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.523140 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 22:26:34.529406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.529380 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:34.543778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.543694 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:34.544116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.544092 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:34.546026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.545656 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:26:34.644368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.644312 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" podStartSLOduration=2.644297486 podStartE2EDuration="2.644297486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:34.642700264 +0000 UTC m=+1230.907693484" watchObservedRunningTime="2026-11-02 22:26:34.644297486 +0000 UTC m=+1230.909290704" Nov 02 22:26:34.667671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.667636 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d219b9b9-b686-4d46-b944-87bb6f9e022b-kube-api-access-sh57s\" (UniqueName: \"kubernetes.io/projected/d219b9b9-b686-4d46-b944-87bb6f9e022b-kube-api-access-sh57s\") pod \"d219b9b9-b686-4d46-b944-87bb6f9e022b\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " Nov 02 22:26:34.667902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.667705 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-openshift-ca\") pod \"d219b9b9-b686-4d46-b944-87bb6f9e022b\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " Nov 02 22:26:34.667973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.667898 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-xks-ca\") pod \"d219b9b9-b686-4d46-b944-87bb6f9e022b\" (UID: \"d219b9b9-b686-4d46-b944-87bb6f9e022b\") " Nov 02 22:26:34.670071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.670034 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-openshift-ca" pod "d219b9b9-b686-4d46-b944-87bb6f9e022b" (UID: "d219b9b9-b686-4d46-b944-87bb6f9e022b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:34.672150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.672119 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d219b9b9-b686-4d46-b944-87bb6f9e022b-kube-api-access-sh57s" pod "d219b9b9-b686-4d46-b944-87bb6f9e022b" (UID: "d219b9b9-b686-4d46-b944-87bb6f9e022b"). InnerVolumeSpecName "kube-api-access-sh57s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:34.672411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.672389 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-xks-ca" pod "d219b9b9-b686-4d46-b944-87bb6f9e022b" (UID: "d219b9b9-b686-4d46-b944-87bb6f9e022b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:34.770682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.770635 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:34.770682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.770672 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh57s\" (UniqueName: \"kubernetes.io/projected/d219b9b9-b686-4d46-b944-87bb6f9e022b-kube-api-access-sh57s\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:34.770682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.770689 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d219b9b9-b686-4d46-b944-87bb6f9e022b-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:34.774135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.774102 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq"] Nov 02 22:26:34.775289 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.775265 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d219b9b9-b686-4d46-b944-87bb6f9e022b" containerName="revoke-keys" Nov 02 22:26:34.775357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.775293 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d219b9b9-b686-4d46-b944-87bb6f9e022b" containerName="revoke-keys" Nov 02 22:26:34.775478 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.775464 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d219b9b9-b686-4d46-b944-87bb6f9e022b" containerName="revoke-keys" Nov 02 22:26:34.779904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.779874 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.840650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.839817 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq"] Nov 02 22:26:34.872336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.872291 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.872547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.872343 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7w6b\" (UniqueName: \"kubernetes.io/projected/70b22b7b-c31f-4568-913a-095eae5a9c8d-kube-api-access-j7w6b\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.873005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.872824 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.974501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.974460 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.974715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.974647 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.974715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.974693 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7w6b\" (UniqueName: \"kubernetes.io/projected/70b22b7b-c31f-4568-913a-095eae5a9c8d-kube-api-access-j7w6b\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.975432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.975401 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:34.977739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:34.977716 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:35.031804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.031763 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96"] Nov 02 22:26:35.037079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.037047 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.041266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.041231 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7w6b\" (UniqueName: \"kubernetes.io/projected/70b22b7b-c31f-4568-913a-095eae5a9c8d-kube-api-access-j7w6b\") pod \"maas-api-revoke-sub-79fc36ea-pzkmq\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:35.094546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.094472 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96"] Nov 02 22:26:35.098220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.098187 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:35.179883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.179843 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.180201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.180175 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctsd\" (UniqueName: \"kubernetes.io/projected/1acce91e-d524-475d-8d70-b488fe67f151-kube-api-access-mctsd\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.180448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.180429 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.284646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.284609 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctsd\" (UniqueName: \"kubernetes.io/projected/1acce91e-d524-475d-8d70-b488fe67f151-kube-api-access-mctsd\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.285347 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.285321 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.286788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.286763 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.287479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.286336 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.291044 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.290746 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.331174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.331129 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" event={"ID":"d219b9b9-b686-4d46-b944-87bb6f9e022b","Type":"ContainerDied","Data":"97720233d2e231d5a4b3a474d1eaa6a8a36623b0b9dcbdd466a62496b387835f"} Nov 02 22:26:35.331352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.331217 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97720233d2e231d5a4b3a474d1eaa6a8a36623b0b9dcbdd466a62496b387835f" Nov 02 22:26:35.331352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.331317 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb" Nov 02 22:26:35.334935 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.334894 5109 generic.go:356] "Generic (PLEG): container finished" podID="c07398f0-55be-42f6-a86e-39d2f96f77fa" containerID="b31427295f0119854dbc58f1282807eae9229d21a279eb0f80c3a3113e312006" exitCode=0 Nov 02 22:26:35.336359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.336325 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" event={"ID":"c07398f0-55be-42f6-a86e-39d2f96f77fa","Type":"ContainerDied","Data":"b31427295f0119854dbc58f1282807eae9229d21a279eb0f80c3a3113e312006"} Nov 02 22:26:35.352115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.352049 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctsd\" (UniqueName: \"kubernetes.io/projected/1acce91e-d524-475d-8d70-b488fe67f151-kube-api-access-mctsd\") pod \"maas-api-revoke-sub-eeecc3bd-whd96\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.380439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.380408 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:35.527852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.527797 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:35.529016 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.528988 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:35.530126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.530096 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:26:35.556313 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:35.556276 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b22b7b_c31f_4568_913a_095eae5a9c8d.slice/crio-1b0518f952c8fd5989df07fdd239d5ef8c5119c1c6699da2eb2f03a96b31899d WatchSource:0}: Error finding container 1b0518f952c8fd5989df07fdd239d5ef8c5119c1c6699da2eb2f03a96b31899d: Status 404 returned error can't find the container with id 1b0518f952c8fd5989df07fdd239d5ef8c5119c1c6699da2eb2f03a96b31899d Nov 02 22:26:35.556603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.556544 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq"] Nov 02 22:26:35.763338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.763312 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96"] Nov 02 22:26:35.911809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:35.911779 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a15ec9-ab64-46b6-abd5-996db792d644" path="/var/lib/kubelet/pods/20a15ec9-ab64-46b6-abd5-996db792d644/volumes" Nov 02 22:26:36.344553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.344507 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" event={"ID":"1acce91e-d524-475d-8d70-b488fe67f151","Type":"ContainerStarted","Data":"8378e2dcccd79f54842fbd0be7f4cce4d7615ec2554c9dd5db37589712585608"} Nov 02 22:26:36.344761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.344655 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" event={"ID":"1acce91e-d524-475d-8d70-b488fe67f151","Type":"ContainerStarted","Data":"3ee821b9e058c7e2d62cdaf563b2934cc7ae75bc5d06cad896e3ec1628162b9a"} Nov 02 22:26:36.346389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.346358 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" event={"ID":"70b22b7b-c31f-4568-913a-095eae5a9c8d","Type":"ContainerStarted","Data":"08e508a60bc3da93fa1a7f639aca97daa78d4e3f79b9f015aee979ea36cf13cf"} Nov 02 22:26:36.346534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.346415 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" event={"ID":"70b22b7b-c31f-4568-913a-095eae5a9c8d","Type":"ContainerStarted","Data":"1b0518f952c8fd5989df07fdd239d5ef8c5119c1c6699da2eb2f03a96b31899d"} Nov 02 22:26:36.459436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.459365 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" podStartSLOduration=2.45934294 podStartE2EDuration="2.45934294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:36.453189315 +0000 UTC m=+1232.718182534" watchObservedRunningTime="2026-11-02 22:26:36.45934294 +0000 UTC m=+1232.724336161" Nov 02 22:26:36.507242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.507204 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:36.563613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.563537 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" podStartSLOduration=2.563521204 podStartE2EDuration="2.563521204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:36.562803436 +0000 UTC m=+1232.827796656" watchObservedRunningTime="2026-11-02 22:26:36.563521204 +0000 UTC m=+1232.828514423" Nov 02 22:26:36.622519 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.622485 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-xks-ca\") pod \"c07398f0-55be-42f6-a86e-39d2f96f77fa\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " Nov 02 22:26:36.622719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.622557 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07398f0-55be-42f6-a86e-39d2f96f77fa-kube-api-access-5kpl9\" (UniqueName: \"kubernetes.io/projected/c07398f0-55be-42f6-a86e-39d2f96f77fa-kube-api-access-5kpl9\") pod \"c07398f0-55be-42f6-a86e-39d2f96f77fa\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " Nov 02 22:26:36.622719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.622652 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-openshift-ca\") pod \"c07398f0-55be-42f6-a86e-39d2f96f77fa\" (UID: \"c07398f0-55be-42f6-a86e-39d2f96f77fa\") " Nov 02 22:26:36.623039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.623004 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-openshift-ca" pod "c07398f0-55be-42f6-a86e-39d2f96f77fa" (UID: "c07398f0-55be-42f6-a86e-39d2f96f77fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:36.623417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.623372 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.624897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.624871 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-xks-ca" pod "c07398f0-55be-42f6-a86e-39d2f96f77fa" (UID: "c07398f0-55be-42f6-a86e-39d2f96f77fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:36.625051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.625032 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07398f0-55be-42f6-a86e-39d2f96f77fa-kube-api-access-5kpl9" pod "c07398f0-55be-42f6-a86e-39d2f96f77fa" (UID: "c07398f0-55be-42f6-a86e-39d2f96f77fa"). InnerVolumeSpecName "kube-api-access-5kpl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:36.725160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.725126 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07398f0-55be-42f6-a86e-39d2f96f77fa-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.725160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:36.725159 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kpl9\" (UniqueName: \"kubernetes.io/projected/c07398f0-55be-42f6-a86e-39d2f96f77fa-kube-api-access-5kpl9\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:37.049868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.049829 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf"] Nov 02 22:26:37.054158 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.054128 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be817925-4cltf"] Nov 02 22:26:37.355926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.355883 5109 generic.go:356] "Generic (PLEG): container finished" podID="1acce91e-d524-475d-8d70-b488fe67f151" containerID="8378e2dcccd79f54842fbd0be7f4cce4d7615ec2554c9dd5db37589712585608" exitCode=0 Nov 02 22:26:37.356141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.355974 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" event={"ID":"1acce91e-d524-475d-8d70-b488fe67f151","Type":"ContainerDied","Data":"8378e2dcccd79f54842fbd0be7f4cce4d7615ec2554c9dd5db37589712585608"} Nov 02 22:26:37.357767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.357746 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" Nov 02 22:26:37.357910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.357778 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp" event={"ID":"c07398f0-55be-42f6-a86e-39d2f96f77fa","Type":"ContainerDied","Data":"4c35f1ec9b45761d5eea3b219a91fbd7611efb7fbb286f3bd9c70b5e619645d7"} Nov 02 22:26:37.357910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.357831 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c35f1ec9b45761d5eea3b219a91fbd7611efb7fbb286f3bd9c70b5e619645d7" Nov 02 22:26:37.359626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.359599 5109 generic.go:356] "Generic (PLEG): container finished" podID="70b22b7b-c31f-4568-913a-095eae5a9c8d" containerID="08e508a60bc3da93fa1a7f639aca97daa78d4e3f79b9f015aee979ea36cf13cf" exitCode=0 Nov 02 22:26:37.359752 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.359625 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" event={"ID":"70b22b7b-c31f-4568-913a-095eae5a9c8d","Type":"ContainerDied","Data":"08e508a60bc3da93fa1a7f639aca97daa78d4e3f79b9f015aee979ea36cf13cf"} Nov 02 22:26:37.894230 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:37.894198 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9d9621-5e45-40f6-bc56-d514fae48378" path="/var/lib/kubelet/pods/1c9d9621-5e45-40f6-bc56-d514fae48378/volumes" Nov 02 22:26:38.553968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.553935 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:38.582531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.582506 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:38.654596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.654490 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-openshift-ca\") pod \"70b22b7b-c31f-4568-913a-095eae5a9c8d\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " Nov 02 22:26:38.654596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.654546 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b22b7b-c31f-4568-913a-095eae5a9c8d-kube-api-access-j7w6b\" (UniqueName: \"kubernetes.io/projected/70b22b7b-c31f-4568-913a-095eae5a9c8d-kube-api-access-j7w6b\") pod \"70b22b7b-c31f-4568-913a-095eae5a9c8d\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " Nov 02 22:26:38.654804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.654721 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-openshift-ca\") pod \"1acce91e-d524-475d-8d70-b488fe67f151\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " Nov 02 22:26:38.654848 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.654808 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-xks-ca\") pod \"70b22b7b-c31f-4568-913a-095eae5a9c8d\" (UID: \"70b22b7b-c31f-4568-913a-095eae5a9c8d\") " Nov 02 22:26:38.654912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.654864 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acce91e-d524-475d-8d70-b488fe67f151-kube-api-access-mctsd\" (UniqueName: \"kubernetes.io/projected/1acce91e-d524-475d-8d70-b488fe67f151-kube-api-access-mctsd\") pod \"1acce91e-d524-475d-8d70-b488fe67f151\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " Nov 02 22:26:38.654978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.654959 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-xks-ca\") pod \"1acce91e-d524-475d-8d70-b488fe67f151\" (UID: \"1acce91e-d524-475d-8d70-b488fe67f151\") " Nov 02 22:26:38.655032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.654997 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-openshift-ca" pod "70b22b7b-c31f-4568-913a-095eae5a9c8d" (UID: "70b22b7b-c31f-4568-913a-095eae5a9c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:38.655121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.655090 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-openshift-ca" pod "1acce91e-d524-475d-8d70-b488fe67f151" (UID: "1acce91e-d524-475d-8d70-b488fe67f151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:38.655925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.655896 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:38.656037 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.655933 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:38.657118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.657093 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b22b7b-c31f-4568-913a-095eae5a9c8d-kube-api-access-j7w6b" pod "70b22b7b-c31f-4568-913a-095eae5a9c8d" (UID: "70b22b7b-c31f-4568-913a-095eae5a9c8d"). InnerVolumeSpecName "kube-api-access-j7w6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:38.657207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.657127 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acce91e-d524-475d-8d70-b488fe67f151-kube-api-access-mctsd" pod "1acce91e-d524-475d-8d70-b488fe67f151" (UID: "1acce91e-d524-475d-8d70-b488fe67f151"). InnerVolumeSpecName "kube-api-access-mctsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:38.657207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.657187 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-xks-ca" pod "70b22b7b-c31f-4568-913a-095eae5a9c8d" (UID: "70b22b7b-c31f-4568-913a-095eae5a9c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:38.657290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.657270 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-xks-ca" pod "1acce91e-d524-475d-8d70-b488fe67f151" (UID: "1acce91e-d524-475d-8d70-b488fe67f151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:38.757298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.757261 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7w6b\" (UniqueName: \"kubernetes.io/projected/70b22b7b-c31f-4568-913a-095eae5a9c8d-kube-api-access-j7w6b\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:38.757298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.757294 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b22b7b-c31f-4568-913a-095eae5a9c8d-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:38.757298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.757306 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mctsd\" (UniqueName: \"kubernetes.io/projected/1acce91e-d524-475d-8d70-b488fe67f151-kube-api-access-mctsd\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:38.757545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:38.757316 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acce91e-d524-475d-8d70-b488fe67f151-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:39.049046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.049012 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc"] Nov 02 22:26:39.053607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.053557 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b63bd6-tfrsc"] Nov 02 22:26:39.377681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.377603 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" Nov 02 22:26:39.377681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.377613 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq" event={"ID":"70b22b7b-c31f-4568-913a-095eae5a9c8d","Type":"ContainerDied","Data":"1b0518f952c8fd5989df07fdd239d5ef8c5119c1c6699da2eb2f03a96b31899d"} Nov 02 22:26:39.377681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.377669 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0518f952c8fd5989df07fdd239d5ef8c5119c1c6699da2eb2f03a96b31899d" Nov 02 22:26:39.379353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.379292 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" Nov 02 22:26:39.379353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.379332 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96" event={"ID":"1acce91e-d524-475d-8d70-b488fe67f151","Type":"ContainerDied","Data":"3ee821b9e058c7e2d62cdaf563b2934cc7ae75bc5d06cad896e3ec1628162b9a"} Nov 02 22:26:39.379523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.379393 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee821b9e058c7e2d62cdaf563b2934cc7ae75bc5d06cad896e3ec1628162b9a" Nov 02 22:26:39.893124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:39.893085 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be01902d-224f-4351-9be1-dc86d62b774a" path="/var/lib/kubelet/pods/be01902d-224f-4351-9be1-dc86d62b774a/volumes" Nov 02 22:26:40.050680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:40.050645 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk"] Nov 02 22:26:40.054798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:40.054770 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ef7566-9sqvk"] Nov 02 22:26:41.893948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:41.893914 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89374f18-0e9b-4fbf-a615-a1f5526c66a6" path="/var/lib/kubelet/pods/89374f18-0e9b-4fbf-a615-a1f5526c66a6/volumes" Nov 02 22:26:42.427501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.427463 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq"] Nov 02 22:26:42.428255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428237 5109 cpu_manager.go:414] "containerMap: removing container" podUID="70b22b7b-c31f-4568-913a-095eae5a9c8d" containerName="revoke-keys" Nov 02 22:26:42.428304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428256 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b22b7b-c31f-4568-913a-095eae5a9c8d" containerName="revoke-keys" Nov 02 22:26:42.428304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428268 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c07398f0-55be-42f6-a86e-39d2f96f77fa" containerName="revoke-keys" Nov 02 22:26:42.428304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428273 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07398f0-55be-42f6-a86e-39d2f96f77fa" containerName="revoke-keys" Nov 02 22:26:42.428304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428281 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1acce91e-d524-475d-8d70-b488fe67f151" containerName="revoke-keys" Nov 02 22:26:42.428304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428287 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acce91e-d524-475d-8d70-b488fe67f151" containerName="revoke-keys" Nov 02 22:26:42.428456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428399 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b22b7b-c31f-4568-913a-095eae5a9c8d" containerName="revoke-keys" Nov 02 22:26:42.428456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428407 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07398f0-55be-42f6-a86e-39d2f96f77fa" containerName="revoke-keys" Nov 02 22:26:42.428456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.428415 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acce91e-d524-475d-8d70-b488fe67f151" containerName="revoke-keys" Nov 02 22:26:42.433800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.433782 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.487836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.487798 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq"] Nov 02 22:26:42.528644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.528594 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:42.528852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.528671 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:26:42.630746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.630703 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-kube-api-access-v7xxs\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.630929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.630841 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.631113 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.631092 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.732705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.732608 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.732914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.732893 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.732986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.732957 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-kube-api-access-v7xxs\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.733675 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.733654 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.735209 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.735187 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.770808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.770768 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb"] Nov 02 22:26:42.775731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.775705 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.791298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.791268 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-kube-api-access-v7xxs\") pod \"maas-api-revoke-sub-2951dd58-rbldq\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:42.841508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.841462 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.841738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.841718 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.841915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.841886 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mb9\" (UniqueName: \"kubernetes.io/projected/3a66a81c-c59c-4fe6-8732-9915fdbc455e-kube-api-access-b9mb9\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.846603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.846557 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb"] Nov 02 22:26:42.943468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.943426 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.943994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.943604 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.943994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.943696 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mb9\" (UniqueName: \"kubernetes.io/projected/3a66a81c-c59c-4fe6-8732-9915fdbc455e-kube-api-access-b9mb9\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.944350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.944323 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:42.946069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:42.946049 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:43.001782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:43.001705 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mb9\" (UniqueName: \"kubernetes.io/projected/3a66a81c-c59c-4fe6-8732-9915fdbc455e-kube-api-access-b9mb9\") pod \"maas-api-revoke-sub-ee26d52d-7s4zb\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:43.049243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:43.049202 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:43.091807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:43.091771 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:43.476592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:43.476167 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq"] Nov 02 22:26:43.523110 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:43.523067 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d3d159_c7a3_45d7_9918_08a13ffc1ec2.slice/crio-ac28c93bfbe0a72dcf257e06e9f13b42dd4a0ec39d691ec2e5d7225cf894f022 WatchSource:0}: Error finding container ac28c93bfbe0a72dcf257e06e9f13b42dd4a0ec39d691ec2e5d7225cf894f022: Status 404 returned error can't find the container with id ac28c93bfbe0a72dcf257e06e9f13b42dd4a0ec39d691ec2e5d7225cf894f022 Nov 02 22:26:43.550084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:43.550026 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb"] Nov 02 22:26:44.671672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:44.663593 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" event={"ID":"3a66a81c-c59c-4fe6-8732-9915fdbc455e","Type":"ContainerStarted","Data":"c5176296a3b21060431aaf8b5599547650be73b1ccf891ba8dad4b1d5964c36b"} Nov 02 22:26:44.671672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:44.666331 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" event={"ID":"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2","Type":"ContainerStarted","Data":"ac28c93bfbe0a72dcf257e06e9f13b42dd4a0ec39d691ec2e5d7225cf894f022"} Nov 02 22:26:45.184816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.184759 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.239889 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.239843 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:26:45.426709 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:26:45.426670 5109 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a66a81c_c59c_4fe6_8732_9915fdbc455e.slice/crio-b58cea898bc3bdec047489d60c0b274c9002c2f601bda9a38bb53fcd347a87c4.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:45.535101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.533986 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.598598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.583698 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.741597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.741479 5109 generic.go:356] "Generic (PLEG): container finished" podID="3a66a81c-c59c-4fe6-8732-9915fdbc455e" containerID="b58cea898bc3bdec047489d60c0b274c9002c2f601bda9a38bb53fcd347a87c4" exitCode=0 Nov 02 22:26:45.745598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.742366 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" event={"ID":"3a66a81c-c59c-4fe6-8732-9915fdbc455e","Type":"ContainerDied","Data":"b58cea898bc3bdec047489d60c0b274c9002c2f601bda9a38bb53fcd347a87c4"} Nov 02 22:26:45.761875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.761835 5109 generic.go:356] "Generic (PLEG): container finished" podID="a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" containerID="6bd5ad4513c81311c606760b7c1c34432ebd87626ed3d2e8ecc9c25f5545d643" exitCode=0 Nov 02 22:26:45.762215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.762186 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" event={"ID":"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2","Type":"ContainerDied","Data":"6bd5ad4513c81311c606760b7c1c34432ebd87626ed3d2e8ecc9c25f5545d643"} Nov 02 22:26:45.810522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.810477 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.859763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.859712 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.924141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:45.924088 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.242731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:46.242673 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.260777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:46.260736 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:26:46.854599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:46.828926 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:26:47.020006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:46.889694 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:26:47.037642 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:46.891512 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:26:47.083597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:46.896472 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:26:47.099653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:46.929513 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.124660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.026799 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:26:47.124660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.116852 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:26:47.124660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.116904 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:26:47.124660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.116946 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:26:47.124660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.116987 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:26:47.205188 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.175823 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Nov 02 22:26:47.254750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.254685 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.459172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.459072 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:47.459445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:47.459412 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.062608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.050360 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.072938 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.072878 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.078622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.074516 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.116088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.116031 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.117248 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.116888 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.256040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.255995 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.412609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.412462 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:48.412883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.412853 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.870077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.869708 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:48.870077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:48.869778 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.094592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:49.078687 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:26:49.094592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:49.078979 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:26:49.094592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:49.079214 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:26:49.174286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:49.127834 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:26:49.174286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:49.130237 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.286594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:49.285706 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.085594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.081965 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.085594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.083138 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:26:50.123616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.086875 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:26:50.190259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.190210 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.191912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.191817 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:26:50.222009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.194655 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:50.222009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.195940 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.270619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.232090 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:50.270619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.232168 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.363657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.345850 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:50.363657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.345917 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.395597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.395206 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:26:50.785320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.785271 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:50.787771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.787711 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.820596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.806757 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:50.820955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.820897 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.903191 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.903053 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 02 22:26:50.920112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:50.919995 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Nov 02 22:26:51.229463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.163231 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:26:51.229463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.163741 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:26:51.229463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.198688 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:51.229463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.198728 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:26:51.229463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.203275 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:51.229463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.205212 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:51.321655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.318119 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:51.321655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.318196 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:51.613892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:51.605871 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:26:52.160994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.148560 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:52.241350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.241230 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:52.305106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.305067 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:52.305430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.305401 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:52.411827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.411803 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:52.414279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.414245 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" Nov 02 22:26:52.418773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.418248 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" Nov 02 22:26:52.467001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.466973 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:26:52.467248 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.467233 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f"] Nov 02 22:26:52.473604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.473544 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" containerName="revoke-keys" Nov 02 22:26:52.473829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.473810 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" containerName="revoke-keys" Nov 02 22:26:52.473966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.473952 5109 cpu_manager.go:414] "containerMap: removing container" podUID="3a66a81c-c59c-4fe6-8732-9915fdbc455e" containerName="revoke-keys" Nov 02 22:26:52.474069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.474050 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a66a81c-c59c-4fe6-8732-9915fdbc455e" containerName="revoke-keys" Nov 02 22:26:52.474462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.474443 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" containerName="revoke-keys" Nov 02 22:26:52.474597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.474582 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a66a81c-c59c-4fe6-8732-9915fdbc455e" containerName="revoke-keys" Nov 02 22:26:52.479408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.479380 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb" event={"ID":"3a66a81c-c59c-4fe6-8732-9915fdbc455e","Type":"ContainerDied","Data":"c5176296a3b21060431aaf8b5599547650be73b1ccf891ba8dad4b1d5964c36b"} Nov 02 22:26:52.479613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.479596 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5176296a3b21060431aaf8b5599547650be73b1ccf891ba8dad4b1d5964c36b" Nov 02 22:26:52.479785 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.479768 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq" event={"ID":"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2","Type":"ContainerDied","Data":"ac28c93bfbe0a72dcf257e06e9f13b42dd4a0ec39d691ec2e5d7225cf894f022"} Nov 02 22:26:52.479891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.479879 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac28c93bfbe0a72dcf257e06e9f13b42dd4a0ec39d691ec2e5d7225cf894f022" Nov 02 22:26:52.480005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.479994 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:26:52.480186 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.480173 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:26:52.480296 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.480286 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f"] Nov 02 22:26:52.480411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.480399 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29"] Nov 02 22:26:52.480551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.480537 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:26:52.480764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.480675 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.480819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.480748 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:26:52.483055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.483035 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:26:52.483196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.483184 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028b1a3f-g4f29"] Nov 02 22:26:52.506606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.506581 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2"] Nov 02 22:26:52.511845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.511817 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.513487 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.513459 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2"] Nov 02 22:26:52.518874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.518847 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-xks-ca\") pod \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " Nov 02 22:26:52.519588 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.519552 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-kube-api-access-v7xxs\") pod \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " Nov 02 22:26:52.519789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.519771 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-openshift-ca\") pod \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " Nov 02 22:26:52.520636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.520612 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-xks-ca\") pod \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\" (UID: \"a0d3d159-c7a3-45d7-9918-08a13ffc1ec2\") " Nov 02 22:26:52.521062 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.521046 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a66a81c-c59c-4fe6-8732-9915fdbc455e-kube-api-access-b9mb9\" (UniqueName: \"kubernetes.io/projected/3a66a81c-c59c-4fe6-8732-9915fdbc455e-kube-api-access-b9mb9\") pod \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " Nov 02 22:26:52.521530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.521395 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-openshift-ca" pod "a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" (UID: "a0d3d159-c7a3-45d7-9918-08a13ffc1ec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:52.521964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.521946 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-openshift-ca\") pod \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\" (UID: \"3a66a81c-c59c-4fe6-8732-9915fdbc455e\") " Nov 02 22:26:52.522582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.522390 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-openshift-ca" pod "3a66a81c-c59c-4fe6-8732-9915fdbc455e" (UID: "3a66a81c-c59c-4fe6-8732-9915fdbc455e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:52.524750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.524719 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqcb\" (UniqueName: \"kubernetes.io/projected/2391038f-56ca-4493-b019-73a01f0ba73a-kube-api-access-9kqcb\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.525720 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.525950 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.525989 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.526065 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ltg\" (UniqueName: \"kubernetes.io/projected/229402a9-b633-4f80-bf48-ea87e221d362-kube-api-access-r5ltg\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.526143 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.526287 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.526305 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.526862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.526341 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-xks-ca" pod "3a66a81c-c59c-4fe6-8732-9915fdbc455e" (UID: "3a66a81c-c59c-4fe6-8732-9915fdbc455e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:52.527701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.527669 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a66a81c-c59c-4fe6-8732-9915fdbc455e-kube-api-access-b9mb9" pod "3a66a81c-c59c-4fe6-8732-9915fdbc455e" (UID: "3a66a81c-c59c-4fe6-8732-9915fdbc455e"). InnerVolumeSpecName "kube-api-access-b9mb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:52.528144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.528118 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-xks-ca" pod "a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" (UID: "a0d3d159-c7a3-45d7-9918-08a13ffc1ec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:52.528349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.528331 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-kube-api-access-v7xxs" pod "a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" (UID: "a0d3d159-c7a3-45d7-9918-08a13ffc1ec2"). InnerVolumeSpecName "kube-api-access-v7xxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.644555 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqcb\" (UniqueName: \"kubernetes.io/projected/2391038f-56ca-4493-b019-73a01f0ba73a-kube-api-access-9kqcb\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.644716 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.644839 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.644926 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.644988 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ltg\" (UniqueName: \"kubernetes.io/projected/229402a9-b633-4f80-bf48-ea87e221d362-kube-api-access-r5ltg\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.645038 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.645172 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.645188 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9mb9\" (UniqueName: \"kubernetes.io/projected/3a66a81c-c59c-4fe6-8732-9915fdbc455e-kube-api-access-b9mb9\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.645204 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a66a81c-c59c-4fe6-8732-9915fdbc455e-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.645590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.645217 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xxs\" (UniqueName: \"kubernetes.io/projected/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2-kube-api-access-v7xxs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.646230 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.646089 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.667597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.659319 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.687600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.670679 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.693331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.693292 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.761636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.760698 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqcb\" (UniqueName: \"kubernetes.io/projected/2391038f-56ca-4493-b019-73a01f0ba73a-kube-api-access-9kqcb\") pod \"maas-api-revoke-sub-4b5ff70e-w8n8f\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.765145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.765074 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ltg\" (UniqueName: \"kubernetes.io/projected/229402a9-b633-4f80-bf48-ea87e221d362-kube-api-access-r5ltg\") pod \"maas-api-revoke-sub-53a95627-zn6j2\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:52.806583 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.806532 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:52.866785 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:52.866754 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:53.202384 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:53.202347 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2391038f_56ca_4493_b019_73a01f0ba73a.slice/crio-bcc23c56605a11fdf4630f22ac1fbeac3aa22a73ce9b9f5736c76b7bdf66a12f WatchSource:0}: Error finding container bcc23c56605a11fdf4630f22ac1fbeac3aa22a73ce9b9f5736c76b7bdf66a12f: Status 404 returned error can't find the container with id bcc23c56605a11fdf4630f22ac1fbeac3aa22a73ce9b9f5736c76b7bdf66a12f Nov 02 22:26:53.203659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:53.203628 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f"] Nov 02 22:26:53.301995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:53.301957 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2"] Nov 02 22:26:53.303744 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:53.303711 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229402a9_b633_4f80_bf48_ea87e221d362.slice/crio-e72507ddb049f6e50803411b1bb921936f06f54a6ae60687038d533ca36ffa27 WatchSource:0}: Error finding container e72507ddb049f6e50803411b1bb921936f06f54a6ae60687038d533ca36ffa27: Status 404 returned error can't find the container with id e72507ddb049f6e50803411b1bb921936f06f54a6ae60687038d533ca36ffa27 Nov 02 22:26:53.429460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:53.429423 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" event={"ID":"229402a9-b633-4f80-bf48-ea87e221d362","Type":"ContainerStarted","Data":"e72507ddb049f6e50803411b1bb921936f06f54a6ae60687038d533ca36ffa27"} Nov 02 22:26:53.430844 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:53.430814 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" event={"ID":"2391038f-56ca-4493-b019-73a01f0ba73a","Type":"ContainerStarted","Data":"bcc23c56605a11fdf4630f22ac1fbeac3aa22a73ce9b9f5736c76b7bdf66a12f"} Nov 02 22:26:53.897639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:53.897606 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3017f9-4e3e-46b3-9000-826e54b789f5" path="/var/lib/kubelet/pods/dc3017f9-4e3e-46b3-9000-826e54b789f5/volumes" Nov 02 22:26:54.049088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.049045 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h"] Nov 02 22:26:54.053448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.053415 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92eb48-gh68h"] Nov 02 22:26:54.441101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.441064 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" event={"ID":"229402a9-b633-4f80-bf48-ea87e221d362","Type":"ContainerStarted","Data":"4c8952d0767f59ba524fa690f67b2a87bda34c7cc2e181499466efcd63fdda7e"} Nov 02 22:26:54.442809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.442781 5109 generic.go:356] "Generic (PLEG): container finished" podID="2391038f-56ca-4493-b019-73a01f0ba73a" containerID="c77c668ec189f285fb298da8d61cd594ab4a859002c839d99cf82f5777de2cdb" exitCode=0 Nov 02 22:26:54.442922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.442845 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" event={"ID":"2391038f-56ca-4493-b019-73a01f0ba73a","Type":"ContainerDied","Data":"c77c668ec189f285fb298da8d61cd594ab4a859002c839d99cf82f5777de2cdb"} Nov 02 22:26:54.552743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.552683 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" podStartSLOduration=2.552665298 podStartE2EDuration="2.552665298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:54.550963742 +0000 UTC m=+1250.815956962" watchObservedRunningTime="2026-11-02 22:26:54.552665298 +0000 UTC m=+1250.817658516" Nov 02 22:26:54.811735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.811697 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp"] Nov 02 22:26:54.816427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.816406 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:54.872182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.872141 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp"] Nov 02 22:26:54.919900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.919864 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18692684-72be-42f9-9704-9acf4bc32081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:54.920106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.920010 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkct7\" (UniqueName: \"kubernetes.io/projected/18692684-72be-42f9-9704-9acf4bc32081-kube-api-access-jkct7\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:54.920200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:54.920175 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18692684-72be-42f9-9704-9acf4bc32081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.021923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.021877 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18692684-72be-42f9-9704-9acf4bc32081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.022100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.022002 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkct7\" (UniqueName: \"kubernetes.io/projected/18692684-72be-42f9-9704-9acf4bc32081-kube-api-access-jkct7\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.022100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.022072 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18692684-72be-42f9-9704-9acf4bc32081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.022522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.022498 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18692684-72be-42f9-9704-9acf4bc32081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.025033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.025000 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18692684-72be-42f9-9704-9acf4bc32081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.080967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.080864 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkct7\" (UniqueName: \"kubernetes.io/projected/18692684-72be-42f9-9704-9acf4bc32081-kube-api-access-jkct7\") pod \"maas-api-revoke-sub-5e44456b-8gwcp\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.132091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.132060 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:55.453172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.453081 5109 generic.go:356] "Generic (PLEG): container finished" podID="229402a9-b633-4f80-bf48-ea87e221d362" containerID="4c8952d0767f59ba524fa690f67b2a87bda34c7cc2e181499466efcd63fdda7e" exitCode=0 Nov 02 22:26:55.453603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.453165 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" event={"ID":"229402a9-b633-4f80-bf48-ea87e221d362","Type":"ContainerDied","Data":"4c8952d0767f59ba524fa690f67b2a87bda34c7cc2e181499466efcd63fdda7e"} Nov 02 22:26:55.509139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.509110 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp"] Nov 02 22:26:55.509812 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:55.509788 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18692684_72be_42f9_9704_9acf4bc32081.slice/crio-a4bec35dcc02e17ddd082df9637bee310304c2a24beb4d0cd358837fc766464d WatchSource:0}: Error finding container a4bec35dcc02e17ddd082df9637bee310304c2a24beb4d0cd358837fc766464d: Status 404 returned error can't find the container with id a4bec35dcc02e17ddd082df9637bee310304c2a24beb4d0cd358837fc766464d Nov 02 22:26:55.623412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.623383 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:55.683969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.683932 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj"] Nov 02 22:26:55.684801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.684779 5109 cpu_manager.go:414] "containerMap: removing container" podUID="2391038f-56ca-4493-b019-73a01f0ba73a" containerName="revoke-keys" Nov 02 22:26:55.684801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.684802 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2391038f-56ca-4493-b019-73a01f0ba73a" containerName="revoke-keys" Nov 02 22:26:55.684985 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.684947 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="2391038f-56ca-4493-b019-73a01f0ba73a" containerName="revoke-keys" Nov 02 22:26:55.688719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.688698 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.734398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.734319 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-openshift-ca\") pod \"2391038f-56ca-4493-b019-73a01f0ba73a\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " Nov 02 22:26:55.734398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.734381 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-xks-ca\") pod \"2391038f-56ca-4493-b019-73a01f0ba73a\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " Nov 02 22:26:55.734670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.734585 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2391038f-56ca-4493-b019-73a01f0ba73a-kube-api-access-9kqcb\" (UniqueName: \"kubernetes.io/projected/2391038f-56ca-4493-b019-73a01f0ba73a-kube-api-access-9kqcb\") pod \"2391038f-56ca-4493-b019-73a01f0ba73a\" (UID: \"2391038f-56ca-4493-b019-73a01f0ba73a\") " Nov 02 22:26:55.735071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.735040 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.735197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.735088 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcpw\" (UniqueName: \"kubernetes.io/projected/15550cb1-ff59-4485-b325-d0016be7c921-kube-api-access-jkcpw\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.735529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.735509 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.735713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.735670 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-openshift-ca" pod "2391038f-56ca-4493-b019-73a01f0ba73a" (UID: "2391038f-56ca-4493-b019-73a01f0ba73a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:55.735977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.735958 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:55.739736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.739707 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-xks-ca" pod "2391038f-56ca-4493-b019-73a01f0ba73a" (UID: "2391038f-56ca-4493-b019-73a01f0ba73a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:55.739736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.739719 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2391038f-56ca-4493-b019-73a01f0ba73a-kube-api-access-9kqcb" pod "2391038f-56ca-4493-b019-73a01f0ba73a" (UID: "2391038f-56ca-4493-b019-73a01f0ba73a"). InnerVolumeSpecName "kube-api-access-9kqcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:55.750460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.750421 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj"] Nov 02 22:26:55.837356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.837317 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.837534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.837483 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.837534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.837507 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcpw\" (UniqueName: \"kubernetes.io/projected/15550cb1-ff59-4485-b325-d0016be7c921-kube-api-access-jkcpw\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.837733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.837698 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2391038f-56ca-4493-b019-73a01f0ba73a-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:55.837733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.837728 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqcb\" (UniqueName: \"kubernetes.io/projected/2391038f-56ca-4493-b019-73a01f0ba73a-kube-api-access-9kqcb\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:55.838138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.838116 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.839988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.839968 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:55.895716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.895680 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4efbcd-b337-4d2f-bf44-e993c14b23a0" path="/var/lib/kubelet/pods/ef4efbcd-b337-4d2f-bf44-e993c14b23a0/volumes" Nov 02 22:26:55.896135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:55.896115 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcpw\" (UniqueName: \"kubernetes.io/projected/15550cb1-ff59-4485-b325-d0016be7c921-kube-api-access-jkcpw\") pod \"maas-api-revoke-sub-12c5a20a-dtngj\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:56.003439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.003354 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:56.384645 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:26:56.384603 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15550cb1_ff59_4485_b325_d0016be7c921.slice/crio-f451eeed4492f0348ab65ca951073b752653a670178b1edcfbd82493ad93f16d WatchSource:0}: Error finding container f451eeed4492f0348ab65ca951073b752653a670178b1edcfbd82493ad93f16d: Status 404 returned error can't find the container with id f451eeed4492f0348ab65ca951073b752653a670178b1edcfbd82493ad93f16d Nov 02 22:26:56.384836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.384817 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj"] Nov 02 22:26:56.465525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.465497 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" Nov 02 22:26:56.466004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.465528 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f" event={"ID":"2391038f-56ca-4493-b019-73a01f0ba73a","Type":"ContainerDied","Data":"bcc23c56605a11fdf4630f22ac1fbeac3aa22a73ce9b9f5736c76b7bdf66a12f"} Nov 02 22:26:56.466004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.465605 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc23c56605a11fdf4630f22ac1fbeac3aa22a73ce9b9f5736c76b7bdf66a12f" Nov 02 22:26:56.467351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.467324 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" event={"ID":"15550cb1-ff59-4485-b325-d0016be7c921","Type":"ContainerStarted","Data":"f451eeed4492f0348ab65ca951073b752653a670178b1edcfbd82493ad93f16d"} Nov 02 22:26:56.469383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.469359 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" event={"ID":"18692684-72be-42f9-9704-9acf4bc32081","Type":"ContainerStarted","Data":"a2e03c865a0477381c46f32212b24b50dae21b99834e7a461eec57419e0c1edc"} Nov 02 22:26:56.469512 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.469401 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" event={"ID":"18692684-72be-42f9-9704-9acf4bc32081","Type":"ContainerStarted","Data":"a4bec35dcc02e17ddd082df9637bee310304c2a24beb4d0cd358837fc766464d"} Nov 02 22:26:56.498848 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:26:56.498807 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:26:56.632022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.631924 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" podStartSLOduration=2.631907199 podStartE2EDuration="2.631907199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:56.628914687 +0000 UTC m=+1252.893907906" watchObservedRunningTime="2026-11-02 22:26:56.631907199 +0000 UTC m=+1252.896900461" Nov 02 22:26:56.654243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.654212 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:56.751625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.751581 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-xks-ca\") pod \"229402a9-b633-4f80-bf48-ea87e221d362\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " Nov 02 22:26:56.751810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.751685 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-openshift-ca\") pod \"229402a9-b633-4f80-bf48-ea87e221d362\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " Nov 02 22:26:56.751904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.751837 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229402a9-b633-4f80-bf48-ea87e221d362-kube-api-access-r5ltg\" (UniqueName: \"kubernetes.io/projected/229402a9-b633-4f80-bf48-ea87e221d362-kube-api-access-r5ltg\") pod \"229402a9-b633-4f80-bf48-ea87e221d362\" (UID: \"229402a9-b633-4f80-bf48-ea87e221d362\") " Nov 02 22:26:56.752067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.752039 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-openshift-ca" pod "229402a9-b633-4f80-bf48-ea87e221d362" (UID: "229402a9-b633-4f80-bf48-ea87e221d362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:56.752588 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.752549 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:56.754070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.754030 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-xks-ca" pod "229402a9-b633-4f80-bf48-ea87e221d362" (UID: "229402a9-b633-4f80-bf48-ea87e221d362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:56.754186 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.754080 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229402a9-b633-4f80-bf48-ea87e221d362-kube-api-access-r5ltg" pod "229402a9-b633-4f80-bf48-ea87e221d362" (UID: "229402a9-b633-4f80-bf48-ea87e221d362"). InnerVolumeSpecName "kube-api-access-r5ltg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:56.854494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.854463 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5ltg\" (UniqueName: \"kubernetes.io/projected/229402a9-b633-4f80-bf48-ea87e221d362-kube-api-access-r5ltg\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:56.854664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:56.854516 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229402a9-b633-4f80-bf48-ea87e221d362-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:57.479635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:57.479535 5109 generic.go:356] "Generic (PLEG): container finished" podID="15550cb1-ff59-4485-b325-d0016be7c921" containerID="bef548f21157debc5e9f51db88666475e0e717470133b04e33f733bff9bb6a5e" exitCode=0 Nov 02 22:26:57.480082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:57.479650 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" event={"ID":"15550cb1-ff59-4485-b325-d0016be7c921","Type":"ContainerDied","Data":"bef548f21157debc5e9f51db88666475e0e717470133b04e33f733bff9bb6a5e"} Nov 02 22:26:57.481273 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:57.481248 5109 generic.go:356] "Generic (PLEG): container finished" podID="18692684-72be-42f9-9704-9acf4bc32081" containerID="a2e03c865a0477381c46f32212b24b50dae21b99834e7a461eec57419e0c1edc" exitCode=0 Nov 02 22:26:57.481417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:57.481368 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" event={"ID":"18692684-72be-42f9-9704-9acf4bc32081","Type":"ContainerDied","Data":"a2e03c865a0477381c46f32212b24b50dae21b99834e7a461eec57419e0c1edc"} Nov 02 22:26:57.482798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:57.482780 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" Nov 02 22:26:57.482888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:57.482783 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2" event={"ID":"229402a9-b633-4f80-bf48-ea87e221d362","Type":"ContainerDied","Data":"e72507ddb049f6e50803411b1bb921936f06f54a6ae60687038d533ca36ffa27"} Nov 02 22:26:57.482923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:57.482887 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e72507ddb049f6e50803411b1bb921936f06f54a6ae60687038d533ca36ffa27" Nov 02 22:26:58.663964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.663933 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:58.686090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.686007 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18692684-72be-42f9-9704-9acf4bc32081-kube-api-access-jkct7\" (UniqueName: \"kubernetes.io/projected/18692684-72be-42f9-9704-9acf4bc32081-kube-api-access-jkct7\") pod \"18692684-72be-42f9-9704-9acf4bc32081\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " Nov 02 22:26:58.686090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.686077 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18692684-72be-42f9-9704-9acf4bc32081-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18692684-72be-42f9-9704-9acf4bc32081-maas-api-xks-ca\") pod \"18692684-72be-42f9-9704-9acf4bc32081\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " Nov 02 22:26:58.686317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.686157 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18692684-72be-42f9-9704-9acf4bc32081-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18692684-72be-42f9-9704-9acf4bc32081-maas-api-openshift-ca\") pod \"18692684-72be-42f9-9704-9acf4bc32081\" (UID: \"18692684-72be-42f9-9704-9acf4bc32081\") " Nov 02 22:26:58.687376 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.686881 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18692684-72be-42f9-9704-9acf4bc32081-maas-api-openshift-ca" pod "18692684-72be-42f9-9704-9acf4bc32081" (UID: "18692684-72be-42f9-9704-9acf4bc32081"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:58.687376 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.687193 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18692684-72be-42f9-9704-9acf4bc32081-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:58.688820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.688785 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18692684-72be-42f9-9704-9acf4bc32081-maas-api-xks-ca" pod "18692684-72be-42f9-9704-9acf4bc32081" (UID: "18692684-72be-42f9-9704-9acf4bc32081"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:58.689421 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.689402 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18692684-72be-42f9-9704-9acf4bc32081-kube-api-access-jkct7" pod "18692684-72be-42f9-9704-9acf4bc32081" (UID: "18692684-72be-42f9-9704-9acf4bc32081"). InnerVolumeSpecName "kube-api-access-jkct7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:58.715344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.715316 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:58.788787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.788750 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-openshift-ca\") pod \"15550cb1-ff59-4485-b325-d0016be7c921\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " Nov 02 22:26:58.788963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.788944 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-xks-ca\") pod \"15550cb1-ff59-4485-b325-d0016be7c921\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " Nov 02 22:26:58.789077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.789060 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15550cb1-ff59-4485-b325-d0016be7c921-kube-api-access-jkcpw\" (UniqueName: \"kubernetes.io/projected/15550cb1-ff59-4485-b325-d0016be7c921-kube-api-access-jkcpw\") pod \"15550cb1-ff59-4485-b325-d0016be7c921\" (UID: \"15550cb1-ff59-4485-b325-d0016be7c921\") " Nov 02 22:26:58.789215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.789185 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-openshift-ca" pod "15550cb1-ff59-4485-b325-d0016be7c921" (UID: "15550cb1-ff59-4485-b325-d0016be7c921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:58.789871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.789844 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:58.790009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.789874 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkct7\" (UniqueName: \"kubernetes.io/projected/18692684-72be-42f9-9704-9acf4bc32081-kube-api-access-jkct7\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:58.790009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.789905 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18692684-72be-42f9-9704-9acf4bc32081-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:58.791336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.791309 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-xks-ca" pod "15550cb1-ff59-4485-b325-d0016be7c921" (UID: "15550cb1-ff59-4485-b325-d0016be7c921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:58.791790 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.791761 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15550cb1-ff59-4485-b325-d0016be7c921-kube-api-access-jkcpw" pod "15550cb1-ff59-4485-b325-d0016be7c921" (UID: "15550cb1-ff59-4485-b325-d0016be7c921"). InnerVolumeSpecName "kube-api-access-jkcpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:58.891020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.890988 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15550cb1-ff59-4485-b325-d0016be7c921-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:58.891020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:58.891019 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcpw\" (UniqueName: \"kubernetes.io/projected/15550cb1-ff59-4485-b325-d0016be7c921-kube-api-access-jkcpw\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:59.048604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.048470 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf"] Nov 02 22:26:59.053891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.053854 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87aa09ae-msgkf"] Nov 02 22:26:59.513400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.513372 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" Nov 02 22:26:59.513619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.513397 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj" event={"ID":"15550cb1-ff59-4485-b325-d0016be7c921","Type":"ContainerDied","Data":"f451eeed4492f0348ab65ca951073b752653a670178b1edcfbd82493ad93f16d"} Nov 02 22:26:59.513619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.513495 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f451eeed4492f0348ab65ca951073b752653a670178b1edcfbd82493ad93f16d" Nov 02 22:26:59.515063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.515038 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" Nov 02 22:26:59.515184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.515038 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp" event={"ID":"18692684-72be-42f9-9704-9acf4bc32081","Type":"ContainerDied","Data":"a4bec35dcc02e17ddd082df9637bee310304c2a24beb4d0cd358837fc766464d"} Nov 02 22:26:59.515184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.515149 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4bec35dcc02e17ddd082df9637bee310304c2a24beb4d0cd358837fc766464d" Nov 02 22:26:59.546252 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:26:59.546216 5109 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18692684_72be_42f9_9704_9acf4bc32081.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15550cb1_ff59_4485_b325_d0016be7c921.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:59.894348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.894267 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4881034b-843a-4e1f-a37d-ba4c37fed87c" path="/var/lib/kubelet/pods/4881034b-843a-4e1f-a37d-ba4c37fed87c/volumes" Nov 02 22:26:59.906247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.906218 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp"] Nov 02 22:26:59.906497 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.906474 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" containerID="cri-o://29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0" gracePeriod=30 Nov 02 22:26:59.910957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.910917 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd"] Nov 02 22:26:59.911167 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.911130 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" containerID="cri-o://019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b" gracePeriod=30 Nov 02 22:26:59.915413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.915391 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm"] Nov 02 22:26:59.915614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:26:59.915587 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" containerID="cri-o://1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc" gracePeriod=30 Nov 02 22:27:00.547733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.547698 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2"] Nov 02 22:27:00.548394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548380 5109 cpu_manager.go:414] "containerMap: removing container" podUID="229402a9-b633-4f80-bf48-ea87e221d362" containerName="revoke-keys" Nov 02 22:27:00.548463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548395 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229402a9-b633-4f80-bf48-ea87e221d362" containerName="revoke-keys" Nov 02 22:27:00.548463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548430 5109 cpu_manager.go:414] "containerMap: removing container" podUID="15550cb1-ff59-4485-b325-d0016be7c921" containerName="revoke-keys" Nov 02 22:27:00.548463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548436 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15550cb1-ff59-4485-b325-d0016be7c921" containerName="revoke-keys" Nov 02 22:27:00.548463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548448 5109 cpu_manager.go:414] "containerMap: removing container" podUID="18692684-72be-42f9-9704-9acf4bc32081" containerName="revoke-keys" Nov 02 22:27:00.548463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548454 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18692684-72be-42f9-9704-9acf4bc32081" containerName="revoke-keys" Nov 02 22:27:00.548652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548557 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="15550cb1-ff59-4485-b325-d0016be7c921" containerName="revoke-keys" Nov 02 22:27:00.548652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548581 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="18692684-72be-42f9-9704-9acf4bc32081" containerName="revoke-keys" Nov 02 22:27:00.548652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.548592 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="229402a9-b633-4f80-bf48-ea87e221d362" containerName="revoke-keys" Nov 02 22:27:00.553976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.553956 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.603672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.603625 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:27:00.603672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.603657 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:27:00.608111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.608081 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2"] Nov 02 22:27:00.620611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.620550 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.620789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.620764 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtvb\" (UniqueName: \"kubernetes.io/projected/f99f5550-aa6f-4571-9b31-f1d964850aac-kube-api-access-cvtvb\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.620843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.620828 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.722742 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.722709 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.722957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.722796 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtvb\" (UniqueName: \"kubernetes.io/projected/f99f5550-aa6f-4571-9b31-f1d964850aac-kube-api-access-cvtvb\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.722957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.722841 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.723555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.723533 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.725339 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.725317 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.780838 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.780805 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtvb\" (UniqueName: \"kubernetes.io/projected/f99f5550-aa6f-4571-9b31-f1d964850aac-kube-api-access-cvtvb\") pod \"maas-api-revoke-sub-e41fff3b-mg8v2\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:00.869851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:00.869760 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:01.262887 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:01.262848 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99f5550_aa6f_4571_9b31_f1d964850aac.slice/crio-17b406bcd47470202e966e498c78d8fd2df43e9f775e8651cabe728569fcb9f2 WatchSource:0}: Error finding container 17b406bcd47470202e966e498c78d8fd2df43e9f775e8651cabe728569fcb9f2: Status 404 returned error can't find the container with id 17b406bcd47470202e966e498c78d8fd2df43e9f775e8651cabe728569fcb9f2 Nov 02 22:27:01.263348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:01.263212 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2"] Nov 02 22:27:01.535077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:01.535034 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" event={"ID":"f99f5550-aa6f-4571-9b31-f1d964850aac","Type":"ContainerStarted","Data":"17b406bcd47470202e966e498c78d8fd2df43e9f775e8651cabe728569fcb9f2"} Nov 02 22:27:02.522050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.522007 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z"] Nov 02 22:27:02.526756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.526732 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.545840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.545804 5109 generic.go:356] "Generic (PLEG): container finished" podID="f99f5550-aa6f-4571-9b31-f1d964850aac" containerID="580ac40d65cdeac290a925c060466066141ae2c418fbfe97697ae44261c085f5" exitCode=0 Nov 02 22:27:02.545985 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.545888 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" event={"ID":"f99f5550-aa6f-4571-9b31-f1d964850aac","Type":"ContainerDied","Data":"580ac40d65cdeac290a925c060466066141ae2c418fbfe97697ae44261c085f5"} Nov 02 22:27:02.582428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.582105 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z"] Nov 02 22:27:02.652383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.652343 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.652624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.652497 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6qj\" (UniqueName: \"kubernetes.io/projected/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-kube-api-access-nq6qj\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.652624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.652527 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.754020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.753980 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6qj\" (UniqueName: \"kubernetes.io/projected/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-kube-api-access-nq6qj\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.754020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.754029 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.754293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.754174 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.754972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.754898 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.756767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.756747 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.812877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.812806 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6qj\" (UniqueName: \"kubernetes.io/projected/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-kube-api-access-nq6qj\") pod \"maas-api-revoke-sub-226e008e-55v7z\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:02.841636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:02.841559 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:03.226232 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:03.226200 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3d9868_5f5a_413a_a60f_fad22d6cb6c5.slice/crio-bdeabeff3c1ef9bf8e56e5abedbe49a1291d6f69ef4ea3ee6613c7558f8201bb WatchSource:0}: Error finding container bdeabeff3c1ef9bf8e56e5abedbe49a1291d6f69ef4ea3ee6613c7558f8201bb: Status 404 returned error can't find the container with id bdeabeff3c1ef9bf8e56e5abedbe49a1291d6f69ef4ea3ee6613c7558f8201bb Nov 02 22:27:03.226470 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.226446 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z"] Nov 02 22:27:03.556832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.556798 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" event={"ID":"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5","Type":"ContainerStarted","Data":"e23e81b27b4c202d829ee9ed9af8cec11d2b959496da2bdf1a1b9cd07c8acc83"} Nov 02 22:27:03.557252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.556862 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" event={"ID":"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5","Type":"ContainerStarted","Data":"bdeabeff3c1ef9bf8e56e5abedbe49a1291d6f69ef4ea3ee6613c7558f8201bb"} Nov 02 22:27:03.672777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.672721 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" podStartSLOduration=1.672704234 podStartE2EDuration="1.672704234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:03.668063389 +0000 UTC m=+1259.933056609" watchObservedRunningTime="2026-11-02 22:27:03.672704234 +0000 UTC m=+1259.937697453" Nov 02 22:27:03.688330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.688302 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:03.768010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.767979 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99f5550-aa6f-4571-9b31-f1d964850aac-kube-api-access-cvtvb\" (UniqueName: \"kubernetes.io/projected/f99f5550-aa6f-4571-9b31-f1d964850aac-kube-api-access-cvtvb\") pod \"f99f5550-aa6f-4571-9b31-f1d964850aac\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " Nov 02 22:27:03.768325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.768293 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-openshift-ca\") pod \"f99f5550-aa6f-4571-9b31-f1d964850aac\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " Nov 02 22:27:03.768600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.768453 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-xks-ca\") pod \"f99f5550-aa6f-4571-9b31-f1d964850aac\" (UID: \"f99f5550-aa6f-4571-9b31-f1d964850aac\") " Nov 02 22:27:03.769010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.768978 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-openshift-ca" pod "f99f5550-aa6f-4571-9b31-f1d964850aac" (UID: "f99f5550-aa6f-4571-9b31-f1d964850aac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:03.769377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.769358 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.770616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.770594 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99f5550-aa6f-4571-9b31-f1d964850aac-kube-api-access-cvtvb" pod "f99f5550-aa6f-4571-9b31-f1d964850aac" (UID: "f99f5550-aa6f-4571-9b31-f1d964850aac"). InnerVolumeSpecName "kube-api-access-cvtvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:03.771221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.771200 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-xks-ca" pod "f99f5550-aa6f-4571-9b31-f1d964850aac" (UID: "f99f5550-aa6f-4571-9b31-f1d964850aac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:03.870960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.870882 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99f5550-aa6f-4571-9b31-f1d964850aac-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.870960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:03.870915 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtvb\" (UniqueName: \"kubernetes.io/projected/f99f5550-aa6f-4571-9b31-f1d964850aac-kube-api-access-cvtvb\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:04.085126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.085082 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-zzqfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:27:04.085335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.085163 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:27:04.293620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.293556 5109 scope.go:122] "RemoveContainer" containerID="fff62f324bb4d1ce38a8bc38d6015b6a9098951b8031e7fdf209a830e9191391" Nov 02 22:27:04.305083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.305061 5109 scope.go:122] "RemoveContainer" containerID="b6a29612d67ee0a3e6306239b619b661fe8eccd3b7718b7ca98eb608723a6abb" Nov 02 22:27:04.315846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.315822 5109 scope.go:122] "RemoveContainer" containerID="fba6a181555709f27295d138c1d6c8669b9dd236d1e37992e0b0f5c5a67bf0be" Nov 02 22:27:04.326155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.326130 5109 scope.go:122] "RemoveContainer" containerID="d77b91bf8d25ba9c7973071a885cead03e5168639a44a31755cc4792ba273d32" Nov 02 22:27:04.337294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.337273 5109 scope.go:122] "RemoveContainer" containerID="1450ec703306243a5019bf884785d17ea9cac6b4d88be91998abe4173c505aab" Nov 02 22:27:04.348781 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.348754 5109 scope.go:122] "RemoveContainer" containerID="7dee7fd063bd7a79eba26000842dedc5119fbf30ff679bbe0a515a3a3d71eb7d" Nov 02 22:27:04.363613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.363548 5109 scope.go:122] "RemoveContainer" containerID="975bb4fce3331e42958c6f50f3c20e0070657b809578df27edfd578c2c739c41" Nov 02 22:27:04.566664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.566555 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" Nov 02 22:27:04.566664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.566555 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2" event={"ID":"f99f5550-aa6f-4571-9b31-f1d964850aac","Type":"ContainerDied","Data":"17b406bcd47470202e966e498c78d8fd2df43e9f775e8651cabe728569fcb9f2"} Nov 02 22:27:04.566664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.566639 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b406bcd47470202e966e498c78d8fd2df43e9f775e8651cabe728569fcb9f2" Nov 02 22:27:04.568450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.568422 5109 generic.go:356] "Generic (PLEG): container finished" podID="7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" containerID="e23e81b27b4c202d829ee9ed9af8cec11d2b959496da2bdf1a1b9cd07c8acc83" exitCode=0 Nov 02 22:27:04.568580 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:04.568459 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" event={"ID":"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5","Type":"ContainerDied","Data":"e23e81b27b4c202d829ee9ed9af8cec11d2b959496da2bdf1a1b9cd07c8acc83"} Nov 02 22:27:05.238945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.238916 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:27:05.272272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.272238 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn"] Nov 02 22:27:05.272998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.272974 5109 cpu_manager.go:414] "containerMap: removing container" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" Nov 02 22:27:05.272998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.272994 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" Nov 02 22:27:05.273155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.273038 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f99f5550-aa6f-4571-9b31-f1d964850aac" containerName="revoke-keys" Nov 02 22:27:05.273155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.273044 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99f5550-aa6f-4571-9b31-f1d964850aac" containerName="revoke-keys" Nov 02 22:27:05.273155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.273149 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerName="istio-proxy" Nov 02 22:27:05.273321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.273168 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99f5550-aa6f-4571-9b31-f1d964850aac" containerName="revoke-keys" Nov 02 22:27:05.277210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.277189 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.277909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.277887 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:27:05.294151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294125 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-kube-api-access-8v94t\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-kube-api-access-8v94t\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294170 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-podinfo\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294252 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-certs\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294347 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-socket\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294474 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-token\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-token\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294548 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c722bd3-a137-4489-9330-037fd83d4d5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c722bd3-a137-4489-9330-037fd83d4d5b-istiod-ca-cert\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294656 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-data\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294697 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-envoy\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294711 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-socket" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.294807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294741 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-credential-socket\") pod \"3c722bd3-a137-4489-9330-037fd83d4d5b\" (UID: \"3c722bd3-a137-4489-9330-037fd83d4d5b\") " Nov 02 22:27:05.294958 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294849 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-certs" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.295117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.295096 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-data" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.295228 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.294955 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c722bd3-a137-4489-9330-037fd83d4d5b-istiod-ca-cert" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:05.295383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.295355 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.295447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.295389 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.295447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.295404 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c722bd3-a137-4489-9330-037fd83d4d5b-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.295447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.295418 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.295447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.295415 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-credential-socket" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.296837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.296814 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-podinfo" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:05.297086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.297067 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-kube-api-access-8v94t" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "kube-api-access-8v94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.297156 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.297113 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-token" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.297283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.297266 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-envoy" pod "3c722bd3-a137-4489-9330-037fd83d4d5b" (UID: "3c722bd3-a137-4489-9330-037fd83d4d5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.297766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.297743 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:27:05.333391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.333354 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn"] Nov 02 22:27:05.396109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396063 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-podinfo\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.396294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396131 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istiod-ca-cert\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.396294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396190 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-envoy\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.396294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396235 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13a2f99-892e-46ca-8ba2-9acd657898bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13a2f99-892e-46ca-8ba2-9acd657898bf-istiod-ca-cert\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.396294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396290 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-data\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.396549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396352 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-kube-api-access-5pm5q\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-kube-api-access-5pm5q\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.396549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396438 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-token\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-token\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.396549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396456 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-socket\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.396549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396497 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-certs\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.396549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396519 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-credential-socket\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.396549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396542 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-token\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-token\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.396983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396549 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istiod-ca-cert" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:05.396983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396560 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-certs\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.396983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396812 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-certs" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.396983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396911 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-socket\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.397176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.396983 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-certs" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.397176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397038 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-credential-socket" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.397275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397194 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-data\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.397275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397257 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-envoy\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.397381 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397300 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-podinfo\") pod \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\" (UID: \"7c6248b1-95a4-4446-8abf-55b4cc516ea7\") " Nov 02 22:27:05.397381 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397335 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-kube-api-access-zf2qc\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-kube-api-access-zf2qc\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.397484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397378 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-credential-socket\") pod \"c13a2f99-892e-46ca-8ba2-9acd657898bf\" (UID: \"c13a2f99-892e-46ca-8ba2-9acd657898bf\") " Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397895 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnkd\" (UniqueName: \"kubernetes.io/projected/f077906d-8b76-4065-81f4-5c2e911266a7-kube-api-access-phnkd\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.397940 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13a2f99-892e-46ca-8ba2-9acd657898bf-istiod-ca-cert" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398141 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398181 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398352 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398368 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c722bd3-a137-4489-9330-037fd83d4d5b-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398383 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v94t\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-kube-api-access-8v94t\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398399 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398413 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398427 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13a2f99-892e-46ca-8ba2-9acd657898bf-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398442 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398456 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398472 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.398623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398485 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c722bd3-a137-4489-9330-037fd83d4d5b-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.399640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.398846 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-socket" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.399640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.399085 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-podinfo" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:05.399640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.399164 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-envoy" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.399640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.399309 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-socket" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.399640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.399322 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-data" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.399640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.399633 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-data" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.399883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.399855 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-credential-socket" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.399938 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.399906 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-token" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.401055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.401033 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-podinfo" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:05.401139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.401078 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-kube-api-access-5pm5q" pod "7c6248b1-95a4-4446-8abf-55b4cc516ea7" (UID: "7c6248b1-95a4-4446-8abf-55b4cc516ea7"). InnerVolumeSpecName "kube-api-access-5pm5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.401777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.401757 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-token" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.402050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.402032 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-envoy" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:05.402093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.402038 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-kube-api-access-zf2qc" pod "c13a2f99-892e-46ca-8ba2-9acd657898bf" (UID: "c13a2f99-892e-46ca-8ba2-9acd657898bf"). InnerVolumeSpecName "kube-api-access-zf2qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.500749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.500652 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnkd\" (UniqueName: \"kubernetes.io/projected/f077906d-8b76-4065-81f4-5c2e911266a7-kube-api-access-phnkd\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.500908 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.500843 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.500908 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.500882 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.501087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501069 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501095 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501111 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501125 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501139 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501153 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2qc\" (UniqueName: \"kubernetes.io/projected/c13a2f99-892e-46ca-8ba2-9acd657898bf-kube-api-access-zf2qc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501170 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501183 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501197 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501212 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13a2f99-892e-46ca-8ba2-9acd657898bf-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501227 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pm5q\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-kube-api-access-5pm5q\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501242 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6248b1-95a4-4446-8abf-55b4cc516ea7-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501255 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6248b1-95a4-4446-8abf-55b4cc516ea7-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.501680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.501520 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.503753 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.503728 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.559350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.559316 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnkd\" (UniqueName: \"kubernetes.io/projected/f077906d-8b76-4065-81f4-5c2e911266a7-kube-api-access-phnkd\") pod \"maas-api-revoke-sub-78e58370-vdcwn\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.579083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.579049 5109 generic.go:356] "Generic (PLEG): container finished" podID="3c722bd3-a137-4489-9330-037fd83d4d5b" containerID="29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0" exitCode=0 Nov 02 22:27:05.579514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.579110 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" Nov 02 22:27:05.579514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.579132 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" event={"ID":"3c722bd3-a137-4489-9330-037fd83d4d5b","Type":"ContainerDied","Data":"29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0"} Nov 02 22:27:05.579514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.579185 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp" event={"ID":"3c722bd3-a137-4489-9330-037fd83d4d5b","Type":"ContainerDied","Data":"7393103bdc0171523b2f081b5e5ab07fcc2271cf3d330549cccfcf18a197784c"} Nov 02 22:27:05.579514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.579207 5109 scope.go:122] "RemoveContainer" containerID="29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0" Nov 02 22:27:05.581041 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.581021 5109 generic.go:356] "Generic (PLEG): container finished" podID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerID="019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b" exitCode=0 Nov 02 22:27:05.581149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.581090 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" Nov 02 22:27:05.581149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.581133 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" event={"ID":"c13a2f99-892e-46ca-8ba2-9acd657898bf","Type":"ContainerDied","Data":"019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b"} Nov 02 22:27:05.581262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.581178 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd" event={"ID":"c13a2f99-892e-46ca-8ba2-9acd657898bf","Type":"ContainerDied","Data":"a28e5edee307e3223971cf48f1cb9d29ff718f29cc0c12ea47e32bd672247776"} Nov 02 22:27:05.582837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.582812 5109 generic.go:356] "Generic (PLEG): container finished" podID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerID="1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc" exitCode=0 Nov 02 22:27:05.582936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.582846 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" event={"ID":"7c6248b1-95a4-4446-8abf-55b4cc516ea7","Type":"ContainerDied","Data":"1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc"} Nov 02 22:27:05.582936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.582873 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" Nov 02 22:27:05.582936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.582896 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm" event={"ID":"7c6248b1-95a4-4446-8abf-55b4cc516ea7","Type":"ContainerDied","Data":"9d115b551d0fcddc77f41ab22ee1350fc843a2f5a7d5c30f7ca20d94157b4616"} Nov 02 22:27:05.604864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.604825 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:05.607955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.605795 5109 scope.go:122] "RemoveContainer" containerID="29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0" Nov 02 22:27:05.607955 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:05.606728 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0\": container with ID starting with 29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0 not found: ID does not exist" containerID="29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0" Nov 02 22:27:05.607955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.606769 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0"} err="failed to get container status \"29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0\": rpc error: code = NotFound desc = could not find container \"29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0\": container with ID starting with 29e1ecc01024b44b82841642d4c9c7df0664a3ad36f5872593bf82706acc29e0 not found: ID does not exist" Nov 02 22:27:05.607955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.606795 5109 scope.go:122] "RemoveContainer" containerID="019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b" Nov 02 22:27:05.667719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.667684 5109 scope.go:122] "RemoveContainer" containerID="019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b" Nov 02 22:27:05.668139 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:05.668099 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b\": container with ID starting with 019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b not found: ID does not exist" containerID="019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b" Nov 02 22:27:05.668271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.668131 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b"} err="failed to get container status \"019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b\": rpc error: code = NotFound desc = could not find container \"019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b\": container with ID starting with 019d806d6b4bf18ded446e83f072ae3b64eb8298c2e14757a6edb5b10303a71b not found: ID does not exist" Nov 02 22:27:05.668271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.668152 5109 scope.go:122] "RemoveContainer" containerID="1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc" Nov 02 22:27:05.695804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.695461 5109 scope.go:122] "RemoveContainer" containerID="1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc" Nov 02 22:27:05.696309 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:05.696278 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc\": container with ID starting with 1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc not found: ID does not exist" containerID="1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc" Nov 02 22:27:05.696419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.696318 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc"} err="failed to get container status \"1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc\": rpc error: code = NotFound desc = could not find container \"1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc\": container with ID starting with 1179ef13848d1236adb2aefbe17c6c66f873f6af2af3aafe989b6b44f2be73bc not found: ID does not exist" Nov 02 22:27:05.826550 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.826522 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:05.830234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.830205 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd"] Nov 02 22:27:05.836402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.836374 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-zzqfd"] Nov 02 22:27:05.893702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.893670 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" path="/var/lib/kubelet/pods/c13a2f99-892e-46ca-8ba2-9acd657898bf/volumes" Nov 02 22:27:05.924273 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.924239 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-xks-ca\") pod \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " Nov 02 22:27:05.924453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.924294 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-kube-api-access-nq6qj\" (UniqueName: \"kubernetes.io/projected/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-kube-api-access-nq6qj\") pod \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " Nov 02 22:27:05.924539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.924452 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-openshift-ca\") pod \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\" (UID: \"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5\") " Nov 02 22:27:05.924894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.924868 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-openshift-ca" pod "7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" (UID: "7c3d9868-5f5a-413a-a60f-fad22d6cb6c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:05.925163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.925144 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.926626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.926599 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-xks-ca" pod "7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" (UID: "7c3d9868-5f5a-413a-a60f-fad22d6cb6c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:05.926744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.926729 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-kube-api-access-nq6qj" pod "7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" (UID: "7c3d9868-5f5a-413a-a60f-fad22d6cb6c5"). InnerVolumeSpecName "kube-api-access-nq6qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.939776 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.939751 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm"] Nov 02 22:27:05.946542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:05.946512 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d6shm"] Nov 02 22:27:06.026258 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.026212 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:06.026258 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.026239 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq6qj\" (UniqueName: \"kubernetes.io/projected/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5-kube-api-access-nq6qj\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:06.052901 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.052869 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp"] Nov 02 22:27:06.053497 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:06.053470 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf077906d_8b76_4065_81f4_5c2e911266a7.slice/crio-cc3fa878659bdc7252237ff21b10f9f448c628276d3b11ed385cd916be03029d WatchSource:0}: Error finding container cc3fa878659bdc7252237ff21b10f9f448c628276d3b11ed385cd916be03029d: Status 404 returned error can't find the container with id cc3fa878659bdc7252237ff21b10f9f448c628276d3b11ed385cd916be03029d Nov 02 22:27:06.058019 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.057991 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn"] Nov 02 22:27:06.065094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.064940 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rclwp"] Nov 02 22:27:06.491027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.490993 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx"] Nov 02 22:27:06.491749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491730 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" containerName="revoke-keys" Nov 02 22:27:06.491749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491746 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" containerName="revoke-keys" Nov 02 22:27:06.491945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491759 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" Nov 02 22:27:06.491945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491765 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" Nov 02 22:27:06.491945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491778 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" Nov 02 22:27:06.491945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491783 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" Nov 02 22:27:06.491945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491890 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" containerName="revoke-keys" Nov 02 22:27:06.491945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491905 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13a2f99-892e-46ca-8ba2-9acd657898bf" containerName="istio-proxy" Nov 02 22:27:06.491945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.491915 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" containerName="istio-proxy" Nov 02 22:27:06.495953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.495930 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.551738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.551704 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx"] Nov 02 22:27:06.595329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.595236 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" event={"ID":"f077906d-8b76-4065-81f4-5c2e911266a7","Type":"ContainerStarted","Data":"f04b848cf5d66bd1a76bfbd49eefc259b8bc783b0dda73d7b3b71d101091b253"} Nov 02 22:27:06.595329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.595305 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" event={"ID":"f077906d-8b76-4065-81f4-5c2e911266a7","Type":"ContainerStarted","Data":"cc3fa878659bdc7252237ff21b10f9f448c628276d3b11ed385cd916be03029d"} Nov 02 22:27:06.598886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.598863 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" event={"ID":"7c3d9868-5f5a-413a-a60f-fad22d6cb6c5","Type":"ContainerDied","Data":"bdeabeff3c1ef9bf8e56e5abedbe49a1291d6f69ef4ea3ee6613c7558f8201bb"} Nov 02 22:27:06.598994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.598891 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z" Nov 02 22:27:06.598994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.598901 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdeabeff3c1ef9bf8e56e5abedbe49a1291d6f69ef4ea3ee6613c7558f8201bb" Nov 02 22:27:06.635794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.635755 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.635994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.635964 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.636148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.636131 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z57p\" (UniqueName: \"kubernetes.io/projected/a513f334-622c-4f92-bad0-09be1fd7f6db-kube-api-access-5z57p\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.717290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.717229 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" podStartSLOduration=1.7172130380000001 podStartE2EDuration="1.717213038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:06.711950938 +0000 UTC m=+1262.976944158" watchObservedRunningTime="2026-11-02 22:27:06.717213038 +0000 UTC m=+1262.982206256" Nov 02 22:27:06.737792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.737753 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.737941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.737885 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.738071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.738054 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z57p\" (UniqueName: \"kubernetes.io/projected/a513f334-622c-4f92-bad0-09be1fd7f6db-kube-api-access-5z57p\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.738457 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.738435 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.740475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.740455 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.796127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.796087 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z57p\" (UniqueName: \"kubernetes.io/projected/a513f334-622c-4f92-bad0-09be1fd7f6db-kube-api-access-5z57p\") pod \"maas-api-revoke-sub-66600c7d-rs5lx\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:06.810859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:06.810824 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:07.193391 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:07.193346 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda513f334_622c_4f92_bad0_09be1fd7f6db.slice/crio-d18d4796872c376e72d8e3186e3709eda737ecb5d7f48f1ca044305a77d70170 WatchSource:0}: Error finding container d18d4796872c376e72d8e3186e3709eda737ecb5d7f48f1ca044305a77d70170: Status 404 returned error can't find the container with id d18d4796872c376e72d8e3186e3709eda737ecb5d7f48f1ca044305a77d70170 Nov 02 22:27:07.195037 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:07.194998 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx"] Nov 02 22:27:07.610402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:07.610361 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" event={"ID":"a513f334-622c-4f92-bad0-09be1fd7f6db","Type":"ContainerStarted","Data":"d18d4796872c376e72d8e3186e3709eda737ecb5d7f48f1ca044305a77d70170"} Nov 02 22:27:07.612227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:07.612199 5109 generic.go:356] "Generic (PLEG): container finished" podID="f077906d-8b76-4065-81f4-5c2e911266a7" containerID="f04b848cf5d66bd1a76bfbd49eefc259b8bc783b0dda73d7b3b71d101091b253" exitCode=0 Nov 02 22:27:07.612337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:07.612278 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" event={"ID":"f077906d-8b76-4065-81f4-5c2e911266a7","Type":"ContainerDied","Data":"f04b848cf5d66bd1a76bfbd49eefc259b8bc783b0dda73d7b3b71d101091b253"} Nov 02 22:27:07.893780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:07.893681 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c722bd3-a137-4489-9330-037fd83d4d5b" path="/var/lib/kubelet/pods/3c722bd3-a137-4489-9330-037fd83d4d5b/volumes" Nov 02 22:27:07.894322 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:07.894297 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6248b1-95a4-4446-8abf-55b4cc516ea7" path="/var/lib/kubelet/pods/7c6248b1-95a4-4446-8abf-55b4cc516ea7/volumes" Nov 02 22:27:08.622284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.622252 5109 generic.go:356] "Generic (PLEG): container finished" podID="a513f334-622c-4f92-bad0-09be1fd7f6db" containerID="df4224617a59da99c388e365ea549aad27b1a520860f823f1ec405700e4040f4" exitCode=0 Nov 02 22:27:08.622757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.622344 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" event={"ID":"a513f334-622c-4f92-bad0-09be1fd7f6db","Type":"ContainerDied","Data":"df4224617a59da99c388e365ea549aad27b1a520860f823f1ec405700e4040f4"} Nov 02 22:27:08.779976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.779950 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:08.870743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.870711 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-xks-ca\") pod \"f077906d-8b76-4065-81f4-5c2e911266a7\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " Nov 02 22:27:08.870895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.870753 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-openshift-ca\") pod \"f077906d-8b76-4065-81f4-5c2e911266a7\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " Nov 02 22:27:08.870895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.870861 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f077906d-8b76-4065-81f4-5c2e911266a7-kube-api-access-phnkd\" (UniqueName: \"kubernetes.io/projected/f077906d-8b76-4065-81f4-5c2e911266a7-kube-api-access-phnkd\") pod \"f077906d-8b76-4065-81f4-5c2e911266a7\" (UID: \"f077906d-8b76-4065-81f4-5c2e911266a7\") " Nov 02 22:27:08.871228 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.871203 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-openshift-ca" pod "f077906d-8b76-4065-81f4-5c2e911266a7" (UID: "f077906d-8b76-4065-81f4-5c2e911266a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:08.871579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.871503 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:08.873084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.873024 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f077906d-8b76-4065-81f4-5c2e911266a7-kube-api-access-phnkd" pod "f077906d-8b76-4065-81f4-5c2e911266a7" (UID: "f077906d-8b76-4065-81f4-5c2e911266a7"). InnerVolumeSpecName "kube-api-access-phnkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:08.873162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.873111 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-xks-ca" pod "f077906d-8b76-4065-81f4-5c2e911266a7" (UID: "f077906d-8b76-4065-81f4-5c2e911266a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:08.973600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.973550 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f077906d-8b76-4065-81f4-5c2e911266a7-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:08.973600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:08.973598 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phnkd\" (UniqueName: \"kubernetes.io/projected/f077906d-8b76-4065-81f4-5c2e911266a7-kube-api-access-phnkd\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:09.048950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.048914 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk"] Nov 02 22:27:09.053241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.053211 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664d0c6d-gzjgk"] Nov 02 22:27:09.632805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.632774 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" Nov 02 22:27:09.632805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.632790 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn" event={"ID":"f077906d-8b76-4065-81f4-5c2e911266a7","Type":"ContainerDied","Data":"cc3fa878659bdc7252237ff21b10f9f448c628276d3b11ed385cd916be03029d"} Nov 02 22:27:09.633328 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.632835 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc3fa878659bdc7252237ff21b10f9f448c628276d3b11ed385cd916be03029d" Nov 02 22:27:09.796316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.796285 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:09.893674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.893591 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6953deba-db7d-44a8-97e9-f356fff60ff4" path="/var/lib/kubelet/pods/6953deba-db7d-44a8-97e9-f356fff60ff4/volumes" Nov 02 22:27:09.987680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.987644 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-openshift-ca\") pod \"a513f334-622c-4f92-bad0-09be1fd7f6db\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " Nov 02 22:27:09.987842 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.987689 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-xks-ca\") pod \"a513f334-622c-4f92-bad0-09be1fd7f6db\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " Nov 02 22:27:09.987968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.987955 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a513f334-622c-4f92-bad0-09be1fd7f6db-kube-api-access-5z57p\" (UniqueName: \"kubernetes.io/projected/a513f334-622c-4f92-bad0-09be1fd7f6db-kube-api-access-5z57p\") pod \"a513f334-622c-4f92-bad0-09be1fd7f6db\" (UID: \"a513f334-622c-4f92-bad0-09be1fd7f6db\") " Nov 02 22:27:09.988142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.988117 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-openshift-ca" pod "a513f334-622c-4f92-bad0-09be1fd7f6db" (UID: "a513f334-622c-4f92-bad0-09be1fd7f6db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:09.988530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.988466 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:09.990164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.990137 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-xks-ca" pod "a513f334-622c-4f92-bad0-09be1fd7f6db" (UID: "a513f334-622c-4f92-bad0-09be1fd7f6db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:09.990261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:09.990214 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a513f334-622c-4f92-bad0-09be1fd7f6db-kube-api-access-5z57p" pod "a513f334-622c-4f92-bad0-09be1fd7f6db" (UID: "a513f334-622c-4f92-bad0-09be1fd7f6db"). InnerVolumeSpecName "kube-api-access-5z57p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:10.090179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:10.090143 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z57p\" (UniqueName: \"kubernetes.io/projected/a513f334-622c-4f92-bad0-09be1fd7f6db-kube-api-access-5z57p\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:10.090179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:10.090183 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a513f334-622c-4f92-bad0-09be1fd7f6db-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:10.642465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:10.642435 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" Nov 02 22:27:10.642976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:10.642480 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx" event={"ID":"a513f334-622c-4f92-bad0-09be1fd7f6db","Type":"ContainerDied","Data":"d18d4796872c376e72d8e3186e3709eda737ecb5d7f48f1ca044305a77d70170"} Nov 02 22:27:10.642976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:10.642531 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d18d4796872c376e72d8e3186e3709eda737ecb5d7f48f1ca044305a77d70170" Nov 02 22:27:11.149219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.149180 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb"] Nov 02 22:27:11.149921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.149904 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f077906d-8b76-4065-81f4-5c2e911266a7" containerName="revoke-keys" Nov 02 22:27:11.149921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.149921 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f077906d-8b76-4065-81f4-5c2e911266a7" containerName="revoke-keys" Nov 02 22:27:11.150030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.149963 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a513f334-622c-4f92-bad0-09be1fd7f6db" containerName="revoke-keys" Nov 02 22:27:11.150030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.149969 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a513f334-622c-4f92-bad0-09be1fd7f6db" containerName="revoke-keys" Nov 02 22:27:11.150117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.150104 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a513f334-622c-4f92-bad0-09be1fd7f6db" containerName="revoke-keys" Nov 02 22:27:11.150167 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.150120 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f077906d-8b76-4065-81f4-5c2e911266a7" containerName="revoke-keys" Nov 02 22:27:11.154284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.154260 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.165222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.165187 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r"] Nov 02 22:27:11.170075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.170049 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205046 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205095 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.205320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205115 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205289 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205349 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205392 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.205531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205463 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.205531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205494 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.205677 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205653 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205693 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8f588d-d144-48c6-8fba-6e6f3a033343-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205801 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.205879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205852 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flf99\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-kube-api-access-flf99\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.205974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.205957 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.206026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.206009 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.206092 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.206072 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50358e22-7a8b-456d-ab7b-41ee21f7c263-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.206148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.206131 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgzc\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-kube-api-access-qkgzc\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.206205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.206177 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.206257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.206220 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.251141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.251106 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb"] Nov 02 22:27:11.301093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.300844 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r"] Nov 02 22:27:11.307522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307491 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.307714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307527 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.307714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307558 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.307714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307620 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.307884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307735 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.307884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307845 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.307983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307884 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8f588d-d144-48c6-8fba-6e6f3a033343-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.307983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307946 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.308078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.307994 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flf99\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-kube-api-access-flf99\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.308078 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.308007 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.308187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.308111 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.308187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.308180 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.308276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.308203 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50358e22-7a8b-456d-ab7b-41ee21f7c263-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.308657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.308630 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.308888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.308861 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.308978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.308949 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.309040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309011 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgzc\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-kube-api-access-qkgzc\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.309091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309058 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.309141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309109 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.309141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309125 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8f588d-d144-48c6-8fba-6e6f3a033343-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.309277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309253 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.309333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309307 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.309384 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309337 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.309437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309414 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.309587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309544 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.309753 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309731 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.309965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.309939 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.310208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.310184 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50358e22-7a8b-456d-ab7b-41ee21f7c263-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.310787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.310741 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.316620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.312554 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.316620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.312612 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.319091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.319069 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.366356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.366327 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.366647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.366616 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.366647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.366636 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flf99\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-kube-api-access-flf99\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.366892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.366869 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgzc\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-kube-api-access-qkgzc\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.470309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.470196 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:11.484030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.483985 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:11.514426 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.514394 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj"] Nov 02 22:27:11.521169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.521140 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.555993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.553118 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm"] Nov 02 22:27:11.559547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.559524 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.585137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.582437 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj"] Nov 02 22:27:11.613354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613318 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.613354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613358 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.613613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613490 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.613613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613534 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.613744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613648 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.613744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613700 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.613812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613754 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.613812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613791 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.613872 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613811 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.613872 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613834 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bjx\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-kube-api-access-t8bjx\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.613872 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613859 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.613993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613886 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.613993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613912 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnxr\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-kube-api-access-cxnxr\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.613993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613964 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.613993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.613982 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.614109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.614005 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.614109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.614063 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.614109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.614097 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.651617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.650593 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm"] Nov 02 22:27:11.714988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.714947 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715003 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715056 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715089 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715126 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bjx\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-kube-api-access-t8bjx\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715165 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715211 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715251 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnxr\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-kube-api-access-cxnxr\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715343 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715372 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715410 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715472 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715467 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715541 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715663 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.715797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715727 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.715797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715782 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715807 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715838 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715879 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715874 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715901 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715948 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.715994 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.716041 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.716097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.716048 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.716490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.716130 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.716613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.716594 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.718491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.718465 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.718646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.718508 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.718646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.718538 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.718718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.718666 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.775646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.775610 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.775829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.775746 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bjx\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-kube-api-access-t8bjx\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.776744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.776722 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.777155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.777134 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnxr\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-kube-api-access-cxnxr\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.846091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.846058 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:11.872397 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:11.872363 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50358e22_7a8b_456d_ab7b_41ee21f7c263.slice/crio-a494ff68966215800105d29761ca9ecda225f06280f83dee6961efb30b59c9fc WatchSource:0}: Error finding container a494ff68966215800105d29761ca9ecda225f06280f83dee6961efb30b59c9fc: Status 404 returned error can't find the container with id a494ff68966215800105d29761ca9ecda225f06280f83dee6961efb30b59c9fc Nov 02 22:27:11.872965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.872943 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb"] Nov 02 22:27:11.875656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.875625 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:11.875755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.875690 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:11.875755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.875733 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:11.892639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.892602 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:11.915462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.915433 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r"] Nov 02 22:27:11.922388 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.922273 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:11.922613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.922557 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:11.923535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.923443 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:11.967877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.967647 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7"] Nov 02 22:27:11.978652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.978628 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg"] Nov 02 22:27:11.978900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.978755 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:11.997033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:11.996980 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.021690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021661 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.021820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021695 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62782404-05cd-4b9c-837c-038514b8c096-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.021820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021717 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75fm\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-kube-api-access-q75fm\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.021820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021746 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.021820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021786 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.021820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021814 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.021983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021843 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.021983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021868 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.021983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.021912 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df8261b-156f-4a46-9155-711018c994b1-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.022178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.022144 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.022211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.022195 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62782404-05cd-4b9c-837c-038514b8c096-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.022244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.022231 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.022541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.022501 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlzn\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-kube-api-access-9jlzn\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.022982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.022956 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7"] Nov 02 22:27:12.023318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.023270 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df8261b-156f-4a46-9155-711018c994b1-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.023418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.023332 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.023418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.023376 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.023760 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.023554 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.037651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.023633 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.073856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.063737 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg"] Nov 02 22:27:12.146970 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.146928 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.147274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.147253 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.147417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.147402 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.147549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.147532 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.147697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.147680 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.148287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.148265 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df8261b-156f-4a46-9155-711018c994b1-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.156770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.156740 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.156993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.156975 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62782404-05cd-4b9c-837c-038514b8c096-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.169150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.169111 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.169475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.169447 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlzn\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-kube-api-access-9jlzn\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.169774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.169751 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df8261b-156f-4a46-9155-711018c994b1-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.169912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.169897 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.170055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.170042 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.170221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.170205 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.182039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.182009 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.203929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.203891 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.204182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.204153 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62782404-05cd-4b9c-837c-038514b8c096-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.204297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.204280 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75fm\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-kube-api-access-q75fm\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.204660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.181915 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.204897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.189662 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df8261b-156f-4a46-9155-711018c994b1-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.205046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.189923 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.205203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.190156 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.205315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.149561 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.205419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.191115 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62782404-05cd-4b9c-837c-038514b8c096-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.205525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.149041 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.206158 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.206099 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.231638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.149346 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.231885 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.149626 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.390420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.388120 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62782404-05cd-4b9c-837c-038514b8c096-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.393359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.392982 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:12.393359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.393083 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:12.393359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.393125 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:12.429368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.412518 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj"] Nov 02 22:27:12.429368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.427993 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75fm\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-kube-api-access-q75fm\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.433854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.433827 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.440582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.439726 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.440582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.440250 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df8261b-156f-4a46-9155-711018c994b1-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.440582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.440510 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.442075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.441636 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlzn\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-kube-api-access-9jlzn\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.442075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.442036 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.540745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.540712 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm"] Nov 02 22:27:12.545648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.545557 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:12.545773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.545706 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:12.545773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.545755 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:12.624436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.624351 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:12.681232 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.680968 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:12.696892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.695029 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" event={"ID":"50358e22-7a8b-456d-ab7b-41ee21f7c263","Type":"ContainerStarted","Data":"68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c"} Nov 02 22:27:12.696892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.695107 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" event={"ID":"50358e22-7a8b-456d-ab7b-41ee21f7c263","Type":"ContainerStarted","Data":"a494ff68966215800105d29761ca9ecda225f06280f83dee6961efb30b59c9fc"} Nov 02 22:27:12.700535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.700484 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" event={"ID":"db8f588d-d144-48c6-8fba-6e6f3a033343","Type":"ContainerStarted","Data":"6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8"} Nov 02 22:27:12.700915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.700807 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" event={"ID":"db8f588d-d144-48c6-8fba-6e6f3a033343","Type":"ContainerStarted","Data":"b6b032c59f75828920f7fc782f47679b8719934084e94cb328c616513788a063"} Nov 02 22:27:12.706763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.705270 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" event={"ID":"eaf2116c-cb61-4b3e-a6dd-e633a53d254b","Type":"ContainerStarted","Data":"bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2"} Nov 02 22:27:12.706763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.705329 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" event={"ID":"eaf2116c-cb61-4b3e-a6dd-e633a53d254b","Type":"ContainerStarted","Data":"607c7d85b44905e3d3f10de95f57a1947fdb992cc302f11411ec99567a92ae0c"} Nov 02 22:27:12.711104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.710730 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" event={"ID":"615227d4-61d6-4f60-91b1-cc5a65a5bd4a","Type":"ContainerStarted","Data":"382b2bae61a928a5c9107d0878632c5da3be9be85d4e28dcdf0797890d6c469c"} Nov 02 22:27:12.825832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.825596 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podStartSLOduration=1.825561803 podStartE2EDuration="1.825561803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:12.820736577 +0000 UTC m=+1269.085729796" watchObservedRunningTime="2026-11-02 22:27:12.825561803 +0000 UTC m=+1269.090555025" Nov 02 22:27:12.846810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.846778 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:12.893166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.893127 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:12.946367 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:12.946304 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podStartSLOduration=1.94627989 podStartE2EDuration="1.94627989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:12.936450602 +0000 UTC m=+1269.201443824" watchObservedRunningTime="2026-11-02 22:27:12.94627989 +0000 UTC m=+1269.211273111" Nov 02 22:27:13.065696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.065627 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podStartSLOduration=2.065604707 podStartE2EDuration="2.065604707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:13.050234987 +0000 UTC m=+1269.315228206" watchObservedRunningTime="2026-11-02 22:27:13.065604707 +0000 UTC m=+1269.330597928" Nov 02 22:27:13.100651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.080832 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7"] Nov 02 22:27:13.101137 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:13.087372 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62782404_05cd_4b9c_837c_038514b8c096.slice/crio-7cc1cddb5312a970fac94424c04f8b12c890b174e64b3d69d566cae63e53a357 WatchSource:0}: Error finding container 7cc1cddb5312a970fac94424c04f8b12c890b174e64b3d69d566cae63e53a357: Status 404 returned error can't find the container with id 7cc1cddb5312a970fac94424c04f8b12c890b174e64b3d69d566cae63e53a357 Nov 02 22:27:13.101137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.090880 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:13.101305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.101204 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:13.101305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.101243 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:13.176838 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:13.176807 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df8261b_156f_4a46_9155_711018c994b1.slice/crio-3accec89dc07e7788bf87f9b89b42bc3189c353e6fcb63036198e846e96c8e95 WatchSource:0}: Error finding container 3accec89dc07e7788bf87f9b89b42bc3189c353e6fcb63036198e846e96c8e95: Status 404 returned error can't find the container with id 3accec89dc07e7788bf87f9b89b42bc3189c353e6fcb63036198e846e96c8e95 Nov 02 22:27:13.181203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.180013 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:13.181203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.180104 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:13.181203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.180146 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:13.184140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.184093 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg"] Nov 02 22:27:13.195267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.195198 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podStartSLOduration=2.195177915 podStartE2EDuration="2.195177915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:13.176727827 +0000 UTC m=+1269.441721094" watchObservedRunningTime="2026-11-02 22:27:13.195177915 +0000 UTC m=+1269.460171135" Nov 02 22:27:13.491858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.491709 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:13.491858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.491778 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:13.494680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.494056 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:13.494680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.494104 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:13.494680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.494185 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:13.494680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.494206 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:13.768616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.765900 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" event={"ID":"615227d4-61d6-4f60-91b1-cc5a65a5bd4a","Type":"ContainerStarted","Data":"1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a"} Nov 02 22:27:13.796599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.795876 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" event={"ID":"9df8261b-156f-4a46-9155-711018c994b1","Type":"ContainerStarted","Data":"5fd5d57c28e636dc0e04f62d3686a04a159996b121a6f0dc81aaa3f0d9ec3d2a"} Nov 02 22:27:13.796599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.795946 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" event={"ID":"9df8261b-156f-4a46-9155-711018c994b1","Type":"ContainerStarted","Data":"3accec89dc07e7788bf87f9b89b42bc3189c353e6fcb63036198e846e96c8e95"} Nov 02 22:27:13.829789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.829739 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" event={"ID":"62782404-05cd-4b9c-837c-038514b8c096","Type":"ContainerStarted","Data":"ab71ee402ba7258a547b5d439db42eeb58654efd01d6c647125b2d47c4c631f2"} Nov 02 22:27:13.830092 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.830066 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" event={"ID":"62782404-05cd-4b9c-837c-038514b8c096","Type":"ContainerStarted","Data":"7cc1cddb5312a970fac94424c04f8b12c890b174e64b3d69d566cae63e53a357"} Nov 02 22:27:13.864608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.863490 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:13.878876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.863556 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:13.973146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.961171 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:13.973458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:13.973421 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:14.176653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.163446 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podStartSLOduration=3.163423736 podStartE2EDuration="3.163423736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:13.941891382 +0000 UTC m=+1270.206884635" watchObservedRunningTime="2026-11-02 22:27:14.163423736 +0000 UTC m=+1270.428416958" Nov 02 22:27:14.198846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.198763 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podStartSLOduration=3.198727979 podStartE2EDuration="3.198727979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:14.141055828 +0000 UTC m=+1270.406049049" watchObservedRunningTime="2026-11-02 22:27:14.198727979 +0000 UTC m=+1270.463721205" Nov 02 22:27:14.471099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.471005 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.471378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.471350 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:14.486007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.485969 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.486297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.486267 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:14.663597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.636678 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:14.696598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.692666 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:14.696598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.692839 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.696598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.692891 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:14.715907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.715864 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.716227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.716191 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:14.897595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.896514 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.897595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.896611 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:14.899298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.899269 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.899497 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:14.899463 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:15.089942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.089895 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:15.090978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.090289 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:27:15.090978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.090808 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:15.134665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.134605 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:15.135413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.135382 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:15.188662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.188529 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:15.322637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.299658 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:15.474153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.473951 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:15.474423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.474386 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:15.496799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.496742 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:15.506194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.505827 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:15.628020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.627980 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:15.628332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.628305 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:15.766660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.765397 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:15.766660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.765458 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:15.897754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.897713 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:15.898359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:15.898326 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:16.170014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.169912 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:16.176490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.176446 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:16.219387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.219347 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:16.223947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.223864 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:27:16.225416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.225384 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:16.295526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.276331 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:27:16.295526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.276961 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:27:16.295526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.277010 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:27:16.295526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.277045 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:27:16.295526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.277082 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:27:16.549632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.536153 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:16.549632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.536225 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:16.554017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.553828 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:16.554334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.554273 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:16.578708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.578231 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:16.578708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.578296 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:16.783778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.783738 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:16.784065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.784036 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:16.836150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.836055 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:16.836412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.836384 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:16.972938 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:16.965775 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.028593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.017746 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.028593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.027977 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:17.028593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.027902 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:17.188840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.188739 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:17.189410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.189380 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Nov 02 22:27:17.227743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.227698 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:17.245216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.245173 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:17.300002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.293711 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:17.370653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.370015 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:17.395794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.395748 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:17.398665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.398019 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:27:17.550723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.550687 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.551006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.550977 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:17.559721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.550733 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.560111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.560036 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:17.884659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.796511 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.884659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.881836 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:17.884659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.832203 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.884659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.881895 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:17.884659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.879755 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.884659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.881937 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:17.997883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:17.997845 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.005593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.001646 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:18.202726 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.202627 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:18.254594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.254259 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:18.309184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.309136 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:18.524046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.370646 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:18.563593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.417417 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:18.613594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.417462 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:18.638596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.536232 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.638596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.614673 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:18.638596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.536310 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:27:18.691070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.675702 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.691399 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.691358 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:18.720836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.692503 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.741638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.724742 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:18.755084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.755045 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.755457 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.755422 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:18.932314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.927857 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.932314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.927927 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:18.932691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.932663 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.932847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:18.932821 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:19.283657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.231345 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:19.309778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.270732 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:19.473198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.473149 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:19.473503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.473461 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:19.490142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.490103 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:19.538182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.538082 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.496498 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.540207 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.608981 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.614660 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.825927 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.663049 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.827327 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.682145 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:19.828749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.827395 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:19.849060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.849005 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:19.849368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.849326 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:19.896093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.896048 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:19.896425 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:19.896390 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:20.317694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.314004 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:20.433800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.433732 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:20.471289 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.471237 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:20.471605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.471538 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:20.515463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.515424 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:20.515783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.515751 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:20.649658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.641703 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:20.649658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.641771 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:20.729628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.698328 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:20.729628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.698388 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:20.738597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.731419 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:20.822846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.822787 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:20.823620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.823544 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:27:20.861614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.828642 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:20.861614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.843939 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:20.861614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.843979 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7216-773e6d"] Nov 02 22:27:20.930756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.844006 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:21.017597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.986012 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:21.031826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.988076 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.032182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.032119 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:21.032385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.031909 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shhv\" (UniqueName: \"kubernetes.io/projected/c53d5200-d456-4a79-b4d8-7c01c0bc3c42-kube-api-access-8shhv\") pod \"test-curl-7216-773e6d\" (UID: \"c53d5200-d456-4a79-b4d8-7c01c0bc3c42\") " pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:21.032693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:20.988388 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.032788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.032720 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:21.165110 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.165079 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-thb7m\"" Nov 02 22:27:21.392622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.354133 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:21.392622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.370947 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shhv\" (UniqueName: \"kubernetes.io/projected/c53d5200-d456-4a79-b4d8-7c01c0bc3c42-kube-api-access-8shhv\") pod \"test-curl-7216-773e6d\" (UID: \"c53d5200-d456-4a79-b4d8-7c01c0bc3c42\") " pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:21.393303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.393260 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7216-773e6d"] Nov 02 22:27:21.443632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.434658 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:21.519594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.519142 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shhv\" (UniqueName: \"kubernetes.io/projected/c53d5200-d456-4a79-b4d8-7c01c0bc3c42-kube-api-access-8shhv\") pod \"test-curl-7216-773e6d\" (UID: \"c53d5200-d456-4a79-b4d8-7c01c0bc3c42\") " pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:21.524633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.521147 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.524925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.524888 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:21.525131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.525113 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.525248 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.525230 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:21.652817 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.652721 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.653063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.653033 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:21.697335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.697294 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.716718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.716672 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:21.716995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.698217 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:21.838624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.824603 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:21.861631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.857954 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.861631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.858015 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:21.862012 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.861972 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:21.862325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.862142 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:21.912709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.912525 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:21.912709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.912601 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:21.945813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:21.945022 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:22.362134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.361681 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:22.436529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.435664 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:22.472143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.472005 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:22.472143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.472076 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:22.507600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.507547 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:22.507907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.507874 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:22.695903 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.694945 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:22.695903 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.695014 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:22.724608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.712097 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:22.724608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.712170 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:22.724608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.712911 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:22.842596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.837676 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:22.951661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.915765 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:22.951661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.916924 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:22.951661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.948090 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:22.951661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.948145 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:22.951661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:22.948297 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:23.044611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.033727 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:23.044611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.033812 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:23.458957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.458914 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:23.486622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.467063 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:23.606414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.606375 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:23.606699 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.606670 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:23.606969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.606950 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:23.607090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.607071 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:23.716964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.671169 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:23.717319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.717281 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:23.717489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.700131 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:23.717630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.717610 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:23.930251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.927142 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7216-773e6d"] Nov 02 22:27:23.936611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.930314 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:23.940592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.937032 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:24.127349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.931852 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:24.127797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.931877 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:24.128206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.931787 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:24.128520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.930484 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:24.128859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.952668 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:24.165587 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.128717 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:24.165895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:23.972542 5109 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:27:24.361852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.361818 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 02 22:27:24.391648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.389291 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7216-773e6d" event={"ID":"c53d5200-d456-4a79-b4d8-7c01c0bc3c42","Type":"ContainerStarted","Data":"aa4603839a16163dd2daabb32805368ed71295c27114953fe56c2c6fc0cf306c"} Nov 02 22:27:24.467136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.467084 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:24.468148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.468113 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:24.476355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.476056 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:24.482748 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.482707 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:24.487993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.487954 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:24.488323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.488294 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:24.636586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.636523 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:24.636906 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.636875 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:24.685789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.685701 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:24.686045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.686018 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:24.849606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.847975 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:24.849606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.848035 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:24.893981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.893945 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:24.894280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:24.894252 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:25.134268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.133068 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.134268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.133953 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.150246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.149253 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.179645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.178764 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:25.183595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.180697 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.480662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.468103 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.698597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.515731 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:27:25.698597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.558884 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:25.698597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.558953 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:25.698597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.559026 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:25.698597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.559049 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:25.698597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.628963 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:25.698597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.629024 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:25.707596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.705812 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:25.707596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.705903 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:25.875635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.872931 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:25.875635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.873003 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:25.923628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.911644 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:25.923628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:25.911707 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:26.138356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.138237 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.186408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.141880 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:26.187030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.166965 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.187764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.179670 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.475128 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.474995 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:26.477115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.477072 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:26.493379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.493336 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:26.541013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.540963 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.544173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.544132 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:26.545939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.545902 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:26.550601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.546813 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:26.550866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.550831 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:26.644597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.640530 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:26.644941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.644880 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:26.681944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.681814 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:26.682453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.682315 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:26.945598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.879454 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:27.026121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.026091 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:27.027659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.960700 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.027916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.027875 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:27.093707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:26.989692 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.093970 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.093936 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:27.181677 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.128681 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Nov 02 22:27:27.202773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.202658 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:27:27.212717 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.212667 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:27.224853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.224797 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:27.225420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.225387 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:27.484656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.477171 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:27.516596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.489857 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.516596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.489920 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:27.516596 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:27.498794 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:27:27.516596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.503127 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.516596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.503201 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:27.566934 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.544734 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:27:27.638603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.637118 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:27.639766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.639734 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.640036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.640008 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:27.745657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.692897 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.775595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.769457 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:27.775595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.720685 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:27.918621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.918291 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.918621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.918354 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:27.918621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.918439 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:27.918621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:27.918464 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:28.207716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.207662 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.262596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.241986 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:28.262596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.242370 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:28.262596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.242681 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:28.516663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.489666 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.516663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.491263 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:28.516663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.491315 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:28.516663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.491398 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:28.516663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.491421 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:28.545978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.545895 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.652596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.649434 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:28.656752 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.656696 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:28.745158 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.744933 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:28.745158 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.745003 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:28.846932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.846834 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:28.847198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.847167 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:28.894075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.894031 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:28.894342 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:28.894310 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:29.217349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.217231 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:27:29.244744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.244683 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.245479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.245428 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:29.246059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.246030 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:29.486474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.486249 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:29.486474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.486316 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:29.487170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.487140 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:29.487397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.487371 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:29.497332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.496325 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.572734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.572672 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.653601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.650645 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:29.653601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.650714 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:29.681725 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.681592 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:29.681725 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.681658 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:29.842447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.841067 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:29.842447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.841134 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.850725 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.850499 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:29.850725 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.850580 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:29.895152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.895012 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:29.895152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:29.895081 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:30.222735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.222618 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.265484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.265432 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.267981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.265895 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:30.267981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.265926 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:30.591595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.567944 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:30.616050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.575322 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.629601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.625460 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:30.633009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.629869 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:30.633262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.632062 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:30.633389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.633365 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:30.649221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.632823 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:30.663788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.663738 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:30.703598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.692721 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:30.708712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.708656 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:30.717720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.712834 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.890219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.870543 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 02 22:27:31.117597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:30.980383 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:31.117597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.116733 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:31.117597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.104704 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:31.117597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.116793 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:31.117597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.106730 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:31.117597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.116846 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:31.224373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.223839 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:31.268880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.268816 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:31.270618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.269309 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:31.272861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.272821 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:31.515657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.509114 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:31.515657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.509175 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:31.531614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.526414 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:31.563679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.544822 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:31.644597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.625622 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:31.644597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.634673 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:31.903657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.874335 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:31.903657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.874425 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:31.909778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.909729 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:31.910075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:31.910046 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:32.226549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.226418 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:32.305683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.292731 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:32.305683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.295323 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:32.305683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.304496 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:32.337415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.337364 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:32.466723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.447082 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:32.502795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.502694 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:32.643719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.565699 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 22:27:32.644058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.644019 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 22:27:32.645718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.645682 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:32.662599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.646956 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:32.698871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.663022 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:32.699469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.687245 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:32.728777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.728720 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:32.729141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.692939 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:32.729355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.729295 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:32.729653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.729629 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:32.733049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.733017 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:32.733299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.732948 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:32.733476 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.733447 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:32.873662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.873624 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.036655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.036532 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:33.036655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.928211 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 02 22:27:33.036941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:32.945117 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.036941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.036711 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:33.228419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.228298 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:33.335932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.335882 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:33.339594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.337351 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:33.339594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.338518 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:33.494874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.494778 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.526808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.526754 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:33.527081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.496450 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:33.527235 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.527210 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:33.686082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.571746 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:33.686082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.571822 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:33.686082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.643542 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.686082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.643641 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:33.732644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.722213 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:33.732644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.722812 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:27:33.732644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.723536 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.732644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.723611 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:33.838877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.838821 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:33.839222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.839173 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:33.855596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.851989 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.855596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.852056 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:33.925120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.925070 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:33.925300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:33.925138 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:34.271606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.259318 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:27:34.344603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.335805 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:34.355602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.352756 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:34.355602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.353157 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:34.420597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.405808 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:34.604651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.596888 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 02 22:27:34.604651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.596967 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 02 22:27:34.647619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.616758 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:34.647619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.616828 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:34.686194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.662901 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:34.686194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.662965 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:34.723944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.723837 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:34.741221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.741176 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:34.818738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.806417 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:34.860151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.860055 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:34.860458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.860434 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:34.875975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:34.860560 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:35.005288 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.005210 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 22:27:35.005836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.005502 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 22:27:35.040165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.039006 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:35.040165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.039069 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:35.240254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.236185 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:35.263581 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.263525 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 22:27:35.263880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.263844 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 22:27:35.287963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.274230 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:35.413413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.412627 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:35.429824 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.429594 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:35.476598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.446825 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:35.500862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.500728 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:35.501184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.501129 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:35.706622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.703727 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:35.706622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.703791 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:35.706622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.704128 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:35.706622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.704180 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:35.725481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.725419 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:27:35.806846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.735811 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:35.806846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.735923 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:35.806846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:35.767017 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:36.280604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.276716 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:36.449504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.449445 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:36.471722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.471632 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:36.553790 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.525670 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:27:36.784815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.632637 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:36.785440 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.699520 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 22:27:36.792683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.699589 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:36.796386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.796347 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:37.003861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.795776 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 22:27:37.004323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.726115 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:37.005090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.770652 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:37.067421 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.795749 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:37.069187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:36.795030 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:37.069878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.069840 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:37.071179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.003439 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:37.071360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.071335 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:37.098937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.096338 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:37.136560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.136513 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:37.140393 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.127010 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Nov 02 22:27:37.305592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.297827 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:27:37.345622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.345209 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:27:37.377430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.377378 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:37.716651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.716120 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:37.716651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.716204 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:37.817093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.813994 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:27:37.821965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:37.797515 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:38.140785 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.140740 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:38.141111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.141080 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:38.199262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.199221 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:38.389619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.341185 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:38.491536 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.491372 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:38.525337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.521714 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:38.550719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.550666 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:38.648596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.600633 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:38.648596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.600715 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:38.714595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.708728 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 22:27:38.714595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.708794 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 22:27:38.821757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.821689 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:27:38.900614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.896037 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:27:39.043820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.043794 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:27:39.053599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.044073 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf"] Nov 02 22:27:39.054333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:38.951172 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:27:39.061257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.061215 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:39.061553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.061520 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:39.191805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.191700 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:39.192074 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.192028 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:39.222598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.207338 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:27:39.222598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.221778 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:39.368325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.368253 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:39.525896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.525860 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf"] Nov 02 22:27:39.526194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.526176 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj"] Nov 02 22:27:39.548113 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.548080 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.562851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.562820 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.563396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.563362 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:39.563532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.563417 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:39.600949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.600783 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj"] Nov 02 22:27:39.603247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.603223 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:27:39.610005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.609977 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:27:39.610150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.604739 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:39.610286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.610263 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:39.641370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.641330 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n"] Nov 02 22:27:39.645200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.644180 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:39.645200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.644237 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:39.659113 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.659079 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9hk\" (UniqueName: \"kubernetes.io/projected/da9a787d-018e-4410-8638-e82beba29e77-kube-api-access-vw9hk\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.659374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.659354 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a787d-018e-4410-8638-e82beba29e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.659732 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.659712 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a787d-018e-4410-8638-e82beba29e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.662293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.662273 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n"] Nov 02 22:27:39.662558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.662543 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.708475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.706899 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-db36b079-openshift-default-dockercfg-6bmjd\"" Nov 02 22:27:39.755420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.755366 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:27:39.786745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.786626 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-token\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.787120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.787093 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.787298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.787277 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istiod-ca-cert\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.787417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.787400 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.787526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.787513 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9hk\" (UniqueName: \"kubernetes.io/projected/da9a787d-018e-4410-8638-e82beba29e77-kube-api-access-vw9hk\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.787666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.787647 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-socket\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.787803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.787784 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh89t\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-kube-api-access-lh89t\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.787919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.787903 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-podinfo\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.788048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.788032 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a787d-018e-4410-8638-e82beba29e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.788229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.788211 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-credential-socket\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.788525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.788505 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n96g\" (UniqueName: \"kubernetes.io/projected/f9039ba1-3275-4eb5-8bb8-756424870347-kube-api-access-4n96g\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.788771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.788749 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-data\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.788975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.788957 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a787d-018e-4410-8638-e82beba29e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.789207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.789189 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-envoy\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.789357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.789342 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-certs\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.809595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.794462 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a787d-018e-4410-8638-e82beba29e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.828436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.828400 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a787d-018e-4410-8638-e82beba29e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.897622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.877389 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9hk\" (UniqueName: \"kubernetes.io/projected/da9a787d-018e-4410-8638-e82beba29e77-kube-api-access-vw9hk\") pod \"maas-api-revoke-sub-29fde503-xf8nf\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.902590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.902364 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-envoy\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.902590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.902524 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-certs\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.902868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.902621 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-token\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.902868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.902718 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.931948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.914917 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:39.931948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.915621 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-certs\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.931948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.916551 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-envoy\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.933700 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istiod-ca-cert\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.933743 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.933777 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-socket\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.933810 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh89t\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-kube-api-access-lh89t\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.933843 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-podinfo\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.933929 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-credential-socket\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.934059 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n96g\" (UniqueName: \"kubernetes.io/projected/f9039ba1-3275-4eb5-8bb8-756424870347-kube-api-access-4n96g\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.935346 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-socket\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.936113 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istiod-ca-cert\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.936859 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:39.965665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.941358 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-credential-socket\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.972011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.967140 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-data\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:39.972011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:39.967755 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-data\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:40.040640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.040214 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-podinfo\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:40.140676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.101961 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:40.140676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.102025 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:40.166461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.165273 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n96g\" (UniqueName: \"kubernetes.io/projected/f9039ba1-3275-4eb5-8bb8-756424870347-kube-api-access-4n96g\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:40.194450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.194416 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:40.194843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.194792 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:40.220740 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.204763 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24"] Nov 02 22:27:40.273445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.272279 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.299512 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.299474 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24"] Nov 02 22:27:40.320722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.320687 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh89t\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-kube-api-access-lh89t\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:40.356461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.356429 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2ab4dd-gbv4n\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:40.390681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.390315 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.390681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.390372 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgb7\" (UniqueName: \"kubernetes.io/projected/2c3f0a39-4389-4545-8bfc-0545b2a0d293-kube-api-access-gpgb7\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.390681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.390429 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.547595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.545008 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.547595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.545069 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgb7\" (UniqueName: \"kubernetes.io/projected/2c3f0a39-4389-4545-8bfc-0545b2a0d293-kube-api-access-gpgb7\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.547595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.545131 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.547595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.547111 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.548635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.548511 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-token\") pod \"e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:40.563840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.563770 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:40.564164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.564117 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:40.597175 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.596847 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:40.647976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.631928 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgb7\" (UniqueName: \"kubernetes.io/projected/2c3f0a39-4389-4545-8bfc-0545b2a0d293-kube-api-access-gpgb7\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.654816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.650778 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:40.654816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.650842 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:40.656021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.655970 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc7b724-d9f24\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.684089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.683302 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:40.684089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.683372 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:40.720520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.719536 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:40.720520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.719623 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:40.836427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.833949 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:40.956594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.942878 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:40.984597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:40.959106 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9"] Nov 02 22:27:41.008656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.008608 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111213 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4j6\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-kube-api-access-7b4j6\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111295 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111334 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111386 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111426 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111556 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111645 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111742 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.112665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.111843 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.145913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.144702 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:41.145913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.144771 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:41.162411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.158328 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:41.164602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.164544 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:27:41.194867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.191540 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9"] Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.214786 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.214871 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.214957 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.215071 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.215110 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4j6\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-kube-api-access-7b4j6\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.215173 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.215213 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.215263 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.215304 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.221441 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.222476 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.223197 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.227711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.223417 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.232740 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.231305 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.250379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.243691 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j"] Nov 02 22:27:41.266141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.265262 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.300747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.297593 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j"] Nov 02 22:27:41.350814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.348942 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4j6\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-kube-api-access-7b4j6\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.354623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.353851 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:41.488593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.461636 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q962\" (UniqueName: \"kubernetes.io/projected/99faa56c-3865-43a8-950b-86c32f17c370-kube-api-access-9q962\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.488593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.461906 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.488593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.461959 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.566100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.564457 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q962\" (UniqueName: \"kubernetes.io/projected/99faa56c-3865-43a8-950b-86c32f17c370-kube-api-access-9q962\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.566100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.564726 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.566100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.564781 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.568138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.567361 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:41.568138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.567413 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:41.568621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.568550 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.622440 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:41.622342 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9a787d_018e_4410_8638_e82beba29e77.slice/crio-02714b8addd6faeb721828c8f3cac7788a8f69107457fced37eac2b10cf4409a WatchSource:0}: Error finding container 02714b8addd6faeb721828c8f3cac7788a8f69107457fced37eac2b10cf4409a: Status 404 returned error can't find the container with id 02714b8addd6faeb721828c8f3cac7788a8f69107457fced37eac2b10cf4409a Nov 02 22:27:41.654954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.653283 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q962\" (UniqueName: \"kubernetes.io/projected/99faa56c-3865-43a8-950b-86c32f17c370-kube-api-access-9q962\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.654954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.653727 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:41.654954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.653782 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:41.691254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.688552 5109 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:41.691254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.690643 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:41.784591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.755964 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:41.784591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.756023 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:41.869987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.869923 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:41.882498 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.878477 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:27:41.922844 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.922802 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:41.923272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.923248 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:41.926210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.923701 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-395e0025-5jn8j\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:41.929101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.929054 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:27:41.930117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.930095 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:27:41.937772 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.937720 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf"] Nov 02 22:27:41.944112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:41.942113 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:42.019317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.019252 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:42.132728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.108713 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:42.172903 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.169761 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:42.212174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.212132 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n"] Nov 02 22:27:42.214818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.214785 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:27:42.299160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.285220 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:42.397397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.397297 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:42.509270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.509188 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:27:42.559400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.559365 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24"] Nov 02 22:27:42.662852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.662823 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j"] Nov 02 22:27:42.694394 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:42.694363 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa74d8b_b126_487a_8ce9_84724a64e4b2.slice/crio-71824a3e3c5ba620e1a8d9c0d9b879a02e01676909b29a4ccd1462f20522635a WatchSource:0}: Error finding container 71824a3e3c5ba620e1a8d9c0d9b879a02e01676909b29a4ccd1462f20522635a: Status 404 returned error can't find the container with id 71824a3e3c5ba620e1a8d9c0d9b879a02e01676909b29a4ccd1462f20522635a Nov 02 22:27:42.697188 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.697151 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:42.697299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.697236 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:42.697299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.697279 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:42.698767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.698742 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj"] Nov 02 22:27:42.874733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.874707 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9"] Nov 02 22:27:42.879993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.879953 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:42.880126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.880047 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:42.880126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:42.880090 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:43.062428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.062396 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" event={"ID":"99faa56c-3865-43a8-950b-86c32f17c370","Type":"ContainerStarted","Data":"060b95b1edbcce6ead8db16b62a8cbec08c52ddd4104b362f58be85ecdca1165"} Nov 02 22:27:43.065808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.065768 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" event={"ID":"f9039ba1-3275-4eb5-8bb8-756424870347","Type":"ContainerStarted","Data":"45b87cfcd154646dcc5efcd27a415e039c2844f9090d60d820d2b919b3417fb5"} Nov 02 22:27:43.068812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.068775 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" event={"ID":"0e17753b-68c5-4a7b-a87b-cd0275a77b1f","Type":"ContainerStarted","Data":"9716992e860f8c5810c606a09e76d4d57c3276d8f97ae3c6a7efc7e78ff0e826"} Nov 02 22:27:43.072133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.072095 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" event={"ID":"3fa74d8b-b126-487a-8ce9-84724a64e4b2","Type":"ContainerStarted","Data":"473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570"} Nov 02 22:27:43.072269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.072185 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" event={"ID":"3fa74d8b-b126-487a-8ce9-84724a64e4b2","Type":"ContainerStarted","Data":"71824a3e3c5ba620e1a8d9c0d9b879a02e01676909b29a4ccd1462f20522635a"} Nov 02 22:27:43.076851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.076816 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" event={"ID":"da9a787d-018e-4410-8638-e82beba29e77","Type":"ContainerStarted","Data":"cd02433086410a9650da2869f53f9f3d7b83201c9f1acb55ff3d449eb82d93fe"} Nov 02 22:27:43.077072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.076879 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" event={"ID":"da9a787d-018e-4410-8638-e82beba29e77","Type":"ContainerStarted","Data":"02714b8addd6faeb721828c8f3cac7788a8f69107457fced37eac2b10cf4409a"} Nov 02 22:27:43.080015 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.079949 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" event={"ID":"2c3f0a39-4389-4545-8bfc-0545b2a0d293","Type":"ContainerStarted","Data":"572dc6abe3dc918732834560fcaedbe487e4a98193926b5d1255d1a80a949b10"} Nov 02 22:27:43.186278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.186164 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" podStartSLOduration=4.186145987 podStartE2EDuration="4.186145987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:43.181561144 +0000 UTC m=+1299.446554364" watchObservedRunningTime="2026-11-02 22:27:43.186145987 +0000 UTC m=+1299.451139203" Nov 02 22:27:43.313189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.313128 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podStartSLOduration=6.313105901 podStartE2EDuration="6.313105901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:43.302467908 +0000 UTC m=+1299.567461127" watchObservedRunningTime="2026-11-02 22:27:43.313105901 +0000 UTC m=+1299.578099122" Nov 02 22:27:43.855868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.846545 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:43.863792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.863761 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:43.987596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:43.987494 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" podStartSLOduration=6.987472257 podStartE2EDuration="6.987472257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:43.421756357 +0000 UTC m=+1299.686749577" watchObservedRunningTime="2026-11-02 22:27:43.987472257 +0000 UTC m=+1300.252465478" Nov 02 22:27:44.093701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.093655 5109 generic.go:356] "Generic (PLEG): container finished" podID="99faa56c-3865-43a8-950b-86c32f17c370" containerID="be2e4d5280433a2047ca5f2c803b1e5f6bd9dc9b2140b535d6148c656a286960" exitCode=0 Nov 02 22:27:44.093878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.093718 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" event={"ID":"99faa56c-3865-43a8-950b-86c32f17c370","Type":"ContainerDied","Data":"be2e4d5280433a2047ca5f2c803b1e5f6bd9dc9b2140b535d6148c656a286960"} Nov 02 22:27:44.096177 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.096113 5109 generic.go:356] "Generic (PLEG): container finished" podID="f9039ba1-3275-4eb5-8bb8-756424870347" containerID="2ec9c1ffc61df737dfda8ed6394e7002cf87afc96a41a4cf60f1409c0603b814" exitCode=0 Nov 02 22:27:44.096321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.096215 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" event={"ID":"f9039ba1-3275-4eb5-8bb8-756424870347","Type":"ContainerDied","Data":"2ec9c1ffc61df737dfda8ed6394e7002cf87afc96a41a4cf60f1409c0603b814"} Nov 02 22:27:44.098402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.098372 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" event={"ID":"0e17753b-68c5-4a7b-a87b-cd0275a77b1f","Type":"ContainerStarted","Data":"2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d"} Nov 02 22:27:44.101179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.101149 5109 generic.go:356] "Generic (PLEG): container finished" podID="da9a787d-018e-4410-8638-e82beba29e77" containerID="cd02433086410a9650da2869f53f9f3d7b83201c9f1acb55ff3d449eb82d93fe" exitCode=0 Nov 02 22:27:44.101333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.101225 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" event={"ID":"da9a787d-018e-4410-8638-e82beba29e77","Type":"ContainerDied","Data":"cd02433086410a9650da2869f53f9f3d7b83201c9f1acb55ff3d449eb82d93fe"} Nov 02 22:27:44.103271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.103246 5109 generic.go:356] "Generic (PLEG): container finished" podID="2c3f0a39-4389-4545-8bfc-0545b2a0d293" containerID="78e8de60ecb069a664ba86fa21b8b046315fc1309cad146b56a1ee65e528374e" exitCode=0 Nov 02 22:27:44.103398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.103312 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" event={"ID":"2c3f0a39-4389-4545-8bfc-0545b2a0d293","Type":"ContainerDied","Data":"78e8de60ecb069a664ba86fa21b8b046315fc1309cad146b56a1ee65e528374e"} Nov 02 22:27:44.103597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.103561 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:44.105010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.104991 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:27:44.297546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.297519 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:44.299427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.299402 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:27:44.299621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.299458 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 22:27:44.840592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:44.834630 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podStartSLOduration=4.83460625 podStartE2EDuration="4.83460625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:44.825612988 +0000 UTC m=+1301.090606208" watchObservedRunningTime="2026-11-02 22:27:44.83460625 +0000 UTC m=+1301.099599466" Nov 02 22:27:45.152037 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.151925 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7216-773e6d" event={"ID":"c53d5200-d456-4a79-b4d8-7c01c0bc3c42","Type":"ContainerStarted","Data":"86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe"} Nov 02 22:27:45.273819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.273758 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7216-773e6d" podStartSLOduration=4.55811749 podStartE2EDuration="25.273739977s" totalImagesPullingTime="20.715622487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:45.267745048 +0000 UTC m=+1301.532738266" watchObservedRunningTime="2026-11-02 22:27:45.273739977 +0000 UTC m=+1301.538733196" Nov 02 22:27:45.297604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.297335 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:27:45.297604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.297411 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 22:27:45.467764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.467342 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:45.513783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.513744 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:45.561711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.561686 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-xks-ca\") pod \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " Nov 02 22:27:45.561880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.561747 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9039ba1-3275-4eb5-8bb8-756424870347-kube-api-access-4n96g\" (UniqueName: \"kubernetes.io/projected/f9039ba1-3275-4eb5-8bb8-756424870347-kube-api-access-4n96g\") pod \"f9039ba1-3275-4eb5-8bb8-756424870347\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " Nov 02 22:27:45.561880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.561796 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-xks-ca\") pod \"f9039ba1-3275-4eb5-8bb8-756424870347\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " Nov 02 22:27:45.561880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.561867 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3f0a39-4389-4545-8bfc-0545b2a0d293-kube-api-access-gpgb7\" (UniqueName: \"kubernetes.io/projected/2c3f0a39-4389-4545-8bfc-0545b2a0d293-kube-api-access-gpgb7\") pod \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " Nov 02 22:27:45.562184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.562167 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-openshift-ca\") pod \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\" (UID: \"2c3f0a39-4389-4545-8bfc-0545b2a0d293\") " Nov 02 22:27:45.562252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.562227 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-openshift-ca\") pod \"f9039ba1-3275-4eb5-8bb8-756424870347\" (UID: \"f9039ba1-3275-4eb5-8bb8-756424870347\") " Nov 02 22:27:45.563310 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.563282 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-openshift-ca" pod "f9039ba1-3275-4eb5-8bb8-756424870347" (UID: "f9039ba1-3275-4eb5-8bb8-756424870347"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:45.563637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.563617 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-openshift-ca" pod "2c3f0a39-4389-4545-8bfc-0545b2a0d293" (UID: "2c3f0a39-4389-4545-8bfc-0545b2a0d293"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:45.566555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.566527 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-xks-ca" pod "2c3f0a39-4389-4545-8bfc-0545b2a0d293" (UID: "2c3f0a39-4389-4545-8bfc-0545b2a0d293"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:45.566842 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.566815 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9039ba1-3275-4eb5-8bb8-756424870347-kube-api-access-4n96g" pod "f9039ba1-3275-4eb5-8bb8-756424870347" (UID: "f9039ba1-3275-4eb5-8bb8-756424870347"). InnerVolumeSpecName "kube-api-access-4n96g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:45.567975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.567948 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-xks-ca" pod "f9039ba1-3275-4eb5-8bb8-756424870347" (UID: "f9039ba1-3275-4eb5-8bb8-756424870347"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:45.569177 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.568672 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3f0a39-4389-4545-8bfc-0545b2a0d293-kube-api-access-gpgb7" pod "2c3f0a39-4389-4545-8bfc-0545b2a0d293" (UID: "2c3f0a39-4389-4545-8bfc-0545b2a0d293"). InnerVolumeSpecName "kube-api-access-gpgb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:45.590294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.590264 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:45.596915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.592726 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:45.664211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.664169 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9a787d-018e-4410-8638-e82beba29e77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a787d-018e-4410-8638-e82beba29e77-maas-api-openshift-ca\") pod \"da9a787d-018e-4410-8638-e82beba29e77\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " Nov 02 22:27:45.664428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.664288 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-xks-ca\") pod \"99faa56c-3865-43a8-950b-86c32f17c370\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " Nov 02 22:27:45.664428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.664325 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9a787d-018e-4410-8638-e82beba29e77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a787d-018e-4410-8638-e82beba29e77-maas-api-xks-ca\") pod \"da9a787d-018e-4410-8638-e82beba29e77\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " Nov 02 22:27:45.664602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.664446 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99faa56c-3865-43a8-950b-86c32f17c370-kube-api-access-9q962\" (UniqueName: \"kubernetes.io/projected/99faa56c-3865-43a8-950b-86c32f17c370-kube-api-access-9q962\") pod \"99faa56c-3865-43a8-950b-86c32f17c370\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " Nov 02 22:27:45.664602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.664538 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9a787d-018e-4410-8638-e82beba29e77-maas-api-openshift-ca" pod "da9a787d-018e-4410-8638-e82beba29e77" (UID: "da9a787d-018e-4410-8638-e82beba29e77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:45.664735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.664559 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-openshift-ca\") pod \"99faa56c-3865-43a8-950b-86c32f17c370\" (UID: \"99faa56c-3865-43a8-950b-86c32f17c370\") " Nov 02 22:27:45.664793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.664761 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9a787d-018e-4410-8638-e82beba29e77-kube-api-access-vw9hk\" (UniqueName: \"kubernetes.io/projected/da9a787d-018e-4410-8638-e82beba29e77-kube-api-access-vw9hk\") pod \"da9a787d-018e-4410-8638-e82beba29e77\" (UID: \"da9a787d-018e-4410-8638-e82beba29e77\") " Nov 02 22:27:45.665898 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.665869 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.666036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.665912 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.666036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.665927 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n96g\" (UniqueName: \"kubernetes.io/projected/f9039ba1-3275-4eb5-8bb8-756424870347-kube-api-access-4n96g\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.666036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.665933 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-openshift-ca" pod "99faa56c-3865-43a8-950b-86c32f17c370" (UID: "99faa56c-3865-43a8-950b-86c32f17c370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:45.666036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.665941 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9039ba1-3275-4eb5-8bb8-756424870347-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.666036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.665984 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpgb7\" (UniqueName: \"kubernetes.io/projected/2c3f0a39-4389-4545-8bfc-0545b2a0d293-kube-api-access-gpgb7\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.666036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.666003 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a787d-018e-4410-8638-e82beba29e77-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.666036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.666018 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f0a39-4389-4545-8bfc-0545b2a0d293-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.667830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.667805 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-xks-ca" pod "99faa56c-3865-43a8-950b-86c32f17c370" (UID: "99faa56c-3865-43a8-950b-86c32f17c370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:45.668319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.668295 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9a787d-018e-4410-8638-e82beba29e77-kube-api-access-vw9hk" pod "da9a787d-018e-4410-8638-e82beba29e77" (UID: "da9a787d-018e-4410-8638-e82beba29e77"). InnerVolumeSpecName "kube-api-access-vw9hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:45.668835 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.668815 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99faa56c-3865-43a8-950b-86c32f17c370-kube-api-access-9q962" pod "99faa56c-3865-43a8-950b-86c32f17c370" (UID: "99faa56c-3865-43a8-950b-86c32f17c370"). InnerVolumeSpecName "kube-api-access-9q962". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:45.670405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.670383 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9a787d-018e-4410-8638-e82beba29e77-maas-api-xks-ca" pod "da9a787d-018e-4410-8638-e82beba29e77" (UID: "da9a787d-018e-4410-8638-e82beba29e77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:45.768641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.768602 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.768641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.768640 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a787d-018e-4410-8638-e82beba29e77-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.768641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.768655 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q962\" (UniqueName: \"kubernetes.io/projected/99faa56c-3865-43a8-950b-86c32f17c370-kube-api-access-9q962\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.768641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.768670 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99faa56c-3865-43a8-950b-86c32f17c370-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:45.768641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:45.768685 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9hk\" (UniqueName: \"kubernetes.io/projected/da9a787d-018e-4410-8638-e82beba29e77-kube-api-access-vw9hk\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.164632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.164602 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" Nov 02 22:27:46.165070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.164602 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24" event={"ID":"2c3f0a39-4389-4545-8bfc-0545b2a0d293","Type":"ContainerDied","Data":"572dc6abe3dc918732834560fcaedbe487e4a98193926b5d1255d1a80a949b10"} Nov 02 22:27:46.165070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.164742 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="572dc6abe3dc918732834560fcaedbe487e4a98193926b5d1255d1a80a949b10" Nov 02 22:27:46.166392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.166374 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" Nov 02 22:27:46.166539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.166395 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j" event={"ID":"99faa56c-3865-43a8-950b-86c32f17c370","Type":"ContainerDied","Data":"060b95b1edbcce6ead8db16b62a8cbec08c52ddd4104b362f58be85ecdca1165"} Nov 02 22:27:46.166539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.166440 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="060b95b1edbcce6ead8db16b62a8cbec08c52ddd4104b362f58be85ecdca1165" Nov 02 22:27:46.168089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.168071 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" Nov 02 22:27:46.168200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.168112 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n" event={"ID":"f9039ba1-3275-4eb5-8bb8-756424870347","Type":"ContainerDied","Data":"45b87cfcd154646dcc5efcd27a415e039c2844f9090d60d820d2b919b3417fb5"} Nov 02 22:27:46.168200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.168164 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45b87cfcd154646dcc5efcd27a415e039c2844f9090d60d820d2b919b3417fb5" Nov 02 22:27:46.169724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.169707 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" Nov 02 22:27:46.169820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.169762 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf" event={"ID":"da9a787d-018e-4410-8638-e82beba29e77","Type":"ContainerDied","Data":"02714b8addd6faeb721828c8f3cac7788a8f69107457fced37eac2b10cf4409a"} Nov 02 22:27:46.169820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.169801 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02714b8addd6faeb721828c8f3cac7788a8f69107457fced37eac2b10cf4409a" Nov 02 22:27:46.297445 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.297411 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:27:46.297691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:46.297471 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 22:27:47.297818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:47.297771 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:27:47.298331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:47.297838 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 22:27:48.298112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.298073 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:27:48.298506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.298139 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 22:27:48.489877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.489834 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc"] Nov 02 22:27:48.490617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490592 5109 cpu_manager.go:414] "containerMap: removing container" podUID="da9a787d-018e-4410-8638-e82beba29e77" containerName="revoke-keys" Nov 02 22:27:48.490617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490610 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9a787d-018e-4410-8638-e82beba29e77" containerName="revoke-keys" Nov 02 22:27:48.490778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490622 5109 cpu_manager.go:414] "containerMap: removing container" podUID="2c3f0a39-4389-4545-8bfc-0545b2a0d293" containerName="revoke-keys" Nov 02 22:27:48.490778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490627 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3f0a39-4389-4545-8bfc-0545b2a0d293" containerName="revoke-keys" Nov 02 22:27:48.490778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490669 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f9039ba1-3275-4eb5-8bb8-756424870347" containerName="revoke-keys" Nov 02 22:27:48.490778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490674 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9039ba1-3275-4eb5-8bb8-756424870347" containerName="revoke-keys" Nov 02 22:27:48.490778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490689 5109 cpu_manager.go:414] "containerMap: removing container" podUID="99faa56c-3865-43a8-950b-86c32f17c370" containerName="revoke-keys" Nov 02 22:27:48.490778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490694 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99faa56c-3865-43a8-950b-86c32f17c370" containerName="revoke-keys" Nov 02 22:27:48.490961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490798 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9a787d-018e-4410-8638-e82beba29e77" containerName="revoke-keys" Nov 02 22:27:48.490961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490805 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3f0a39-4389-4545-8bfc-0545b2a0d293" containerName="revoke-keys" Nov 02 22:27:48.490961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490820 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9039ba1-3275-4eb5-8bb8-756424870347" containerName="revoke-keys" Nov 02 22:27:48.490961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.490828 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="99faa56c-3865-43a8-950b-86c32f17c370" containerName="revoke-keys" Nov 02 22:27:48.496363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.496340 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.557217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.557141 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc"] Nov 02 22:27:48.563319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.563288 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.563500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.563364 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.563542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.563485 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgzm\" (UniqueName: \"kubernetes.io/projected/53060a2e-b077-4b08-a8db-704187f0e105-kube-api-access-2rgzm\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.594002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.593969 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:27:48.594183 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.594073 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:27:48.664949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.664907 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.665387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.664987 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.665387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.665030 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgzm\" (UniqueName: \"kubernetes.io/projected/53060a2e-b077-4b08-a8db-704187f0e105-kube-api-access-2rgzm\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.665701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.665675 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.667660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.667640 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.751756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.751714 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgzm\" (UniqueName: \"kubernetes.io/projected/53060a2e-b077-4b08-a8db-704187f0e105-kube-api-access-2rgzm\") pod \"maas-api-revoke-sub-c8ada716-28mwc\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:48.811682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:48.811558 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:49.220846 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:49.220803 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53060a2e_b077_4b08_a8db_704187f0e105.slice/crio-b71ad0989345603f0158853fe1d3e4ab40b399619f7b2b590419ea2c10a996d8 WatchSource:0}: Error finding container b71ad0989345603f0158853fe1d3e4ab40b399619f7b2b590419ea2c10a996d8: Status 404 returned error can't find the container with id b71ad0989345603f0158853fe1d3e4ab40b399619f7b2b590419ea2c10a996d8 Nov 02 22:27:49.246349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.245682 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc"] Nov 02 22:27:49.297633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.297584 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:27:49.297786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.297653 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 22:27:49.959691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.959597 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx"] Nov 02 22:27:49.964114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.964087 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:49.991415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.991372 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hnn\" (UniqueName: \"kubernetes.io/projected/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-kube-api-access-c7hnn\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:49.991606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.991502 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:49.991676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:49.991658 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.021351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.021308 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx"] Nov 02 22:27:50.093365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.093332 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hnn\" (UniqueName: \"kubernetes.io/projected/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-kube-api-access-c7hnn\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.093633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.093402 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.093633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.093451 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.094348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.094311 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.096248 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.096228 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.155205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.155175 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hnn\" (UniqueName: \"kubernetes.io/projected/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-kube-api-access-c7hnn\") pod \"maas-api-revoke-sub-bf43f723-vx8vx\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.209434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.209387 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" event={"ID":"53060a2e-b077-4b08-a8db-704187f0e105","Type":"ContainerStarted","Data":"55810f79d9fb9aba2610196bcb5de117478dbda0870cfc43ec509d22d84300aa"} Nov 02 22:27:50.209635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.209465 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" event={"ID":"53060a2e-b077-4b08-a8db-704187f0e105","Type":"ContainerStarted","Data":"b71ad0989345603f0158853fe1d3e4ab40b399619f7b2b590419ea2c10a996d8"} Nov 02 22:27:50.285234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.285191 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:50.327797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.327724 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" podStartSLOduration=2.327405029 podStartE2EDuration="2.327405029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:50.325249793 +0000 UTC m=+1306.590243016" watchObservedRunningTime="2026-11-02 22:27:50.327405029 +0000 UTC m=+1306.592398249" Nov 02 22:27:50.721466 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:50.721429 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6e68c5_3cd1_4202_a238_e10d85c61a6d.slice/crio-b0a2270ab6fc94932bdf40c9cd108fdff3687be93c2f56b3d3140493ba702acf WatchSource:0}: Error finding container b0a2270ab6fc94932bdf40c9cd108fdff3687be93c2f56b3d3140493ba702acf: Status 404 returned error can't find the container with id b0a2270ab6fc94932bdf40c9cd108fdff3687be93c2f56b3d3140493ba702acf Nov 02 22:27:50.723826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.723737 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx"] Nov 02 22:27:50.766290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.766254 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:50.767216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.767137 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:50.768213 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:50.768102 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:27:51.091275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.091243 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv"] Nov 02 22:27:51.097640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.097615 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.114125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.114093 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.114280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.114201 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ws5x\" (UniqueName: \"kubernetes.io/projected/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-kube-api-access-4ws5x\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.114418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.114401 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.155886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.155847 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv"] Nov 02 22:27:51.215993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.215958 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.216215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.216194 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.216286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.216269 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ws5x\" (UniqueName: \"kubernetes.io/projected/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-kube-api-access-4ws5x\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.217319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.217292 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.219471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.219440 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.227007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.226965 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" event={"ID":"0b6e68c5-3cd1-4202-a238-e10d85c61a6d","Type":"ContainerStarted","Data":"b0a2270ab6fc94932bdf40c9cd108fdff3687be93c2f56b3d3140493ba702acf"} Nov 02 22:27:51.229501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.229462 5109 generic.go:356] "Generic (PLEG): container finished" podID="53060a2e-b077-4b08-a8db-704187f0e105" containerID="55810f79d9fb9aba2610196bcb5de117478dbda0870cfc43ec509d22d84300aa" exitCode=0 Nov 02 22:27:51.229688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.229633 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" event={"ID":"53060a2e-b077-4b08-a8db-704187f0e105","Type":"ContainerDied","Data":"55810f79d9fb9aba2610196bcb5de117478dbda0870cfc43ec509d22d84300aa"} Nov 02 22:27:51.311537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.311498 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ws5x\" (UniqueName: \"kubernetes.io/projected/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-kube-api-access-4ws5x\") pod \"maas-api-revoke-sub-af46f522-vxtkv\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.413419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.413307 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:51.811836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:51.810767 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv"] Nov 02 22:27:52.243142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.243104 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" event={"ID":"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c","Type":"ContainerStarted","Data":"084b024f77d2a1fa7f25e8acbc8dcc89da957f0c77d52ff1ade416858685b1bd"} Nov 02 22:27:52.245026 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.245002 5109 generic.go:356] "Generic (PLEG): container finished" podID="0b6e68c5-3cd1-4202-a238-e10d85c61a6d" containerID="7ac664758e1cdcbbe5d5f6c943b9d508689b15f6663b0245a1dca8c999b46b4c" exitCode=0 Nov 02 22:27:52.245143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.245094 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" event={"ID":"0b6e68c5-3cd1-4202-a238-e10d85c61a6d","Type":"ContainerDied","Data":"7ac664758e1cdcbbe5d5f6c943b9d508689b15f6663b0245a1dca8c999b46b4c"} Nov 02 22:27:52.420472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.420442 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:52.446635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.446593 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53060a2e-b077-4b08-a8db-704187f0e105-kube-api-access-2rgzm\" (UniqueName: \"kubernetes.io/projected/53060a2e-b077-4b08-a8db-704187f0e105-kube-api-access-2rgzm\") pod \"53060a2e-b077-4b08-a8db-704187f0e105\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " Nov 02 22:27:52.446816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.446740 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-openshift-ca\") pod \"53060a2e-b077-4b08-a8db-704187f0e105\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " Nov 02 22:27:52.446924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.446906 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-xks-ca\") pod \"53060a2e-b077-4b08-a8db-704187f0e105\" (UID: \"53060a2e-b077-4b08-a8db-704187f0e105\") " Nov 02 22:27:52.447151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.447121 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-openshift-ca" pod "53060a2e-b077-4b08-a8db-704187f0e105" (UID: "53060a2e-b077-4b08-a8db-704187f0e105"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:52.447531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.447514 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:52.448864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.448844 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53060a2e-b077-4b08-a8db-704187f0e105-kube-api-access-2rgzm" pod "53060a2e-b077-4b08-a8db-704187f0e105" (UID: "53060a2e-b077-4b08-a8db-704187f0e105"). InnerVolumeSpecName "kube-api-access-2rgzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:52.448982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.448970 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-xks-ca" pod "53060a2e-b077-4b08-a8db-704187f0e105" (UID: "53060a2e-b077-4b08-a8db-704187f0e105"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:52.550044 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.550003 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53060a2e-b077-4b08-a8db-704187f0e105-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:52.550044 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:52.550044 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgzm\" (UniqueName: \"kubernetes.io/projected/53060a2e-b077-4b08-a8db-704187f0e105-kube-api-access-2rgzm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:53.256854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.256819 5109 generic.go:356] "Generic (PLEG): container finished" podID="02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" containerID="43c4115927254ab0cd9b27e4a9d57dbf60eed10a7c0b38bb490d93e1f21c99cc" exitCode=0 Nov 02 22:27:53.257335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.256908 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" event={"ID":"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c","Type":"ContainerDied","Data":"43c4115927254ab0cd9b27e4a9d57dbf60eed10a7c0b38bb490d93e1f21c99cc"} Nov 02 22:27:53.258746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.258727 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" Nov 02 22:27:53.258849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.258758 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc" event={"ID":"53060a2e-b077-4b08-a8db-704187f0e105","Type":"ContainerDied","Data":"b71ad0989345603f0158853fe1d3e4ab40b399619f7b2b590419ea2c10a996d8"} Nov 02 22:27:53.258849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.258798 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b71ad0989345603f0158853fe1d3e4ab40b399619f7b2b590419ea2c10a996d8" Nov 02 22:27:53.436077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.436040 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:53.565754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.565649 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-openshift-ca\") pod \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " Nov 02 22:27:53.565754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.565697 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-xks-ca\") pod \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " Nov 02 22:27:53.565754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.565733 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-kube-api-access-c7hnn\" (UniqueName: \"kubernetes.io/projected/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-kube-api-access-c7hnn\") pod \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\" (UID: \"0b6e68c5-3cd1-4202-a238-e10d85c61a6d\") " Nov 02 22:27:53.566121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.566092 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-openshift-ca" pod "0b6e68c5-3cd1-4202-a238-e10d85c61a6d" (UID: "0b6e68c5-3cd1-4202-a238-e10d85c61a6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:53.566415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.566396 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:53.568135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.568113 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-xks-ca" pod "0b6e68c5-3cd1-4202-a238-e10d85c61a6d" (UID: "0b6e68c5-3cd1-4202-a238-e10d85c61a6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:53.568198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.568160 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-kube-api-access-c7hnn" pod "0b6e68c5-3cd1-4202-a238-e10d85c61a6d" (UID: "0b6e68c5-3cd1-4202-a238-e10d85c61a6d"). InnerVolumeSpecName "kube-api-access-c7hnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:53.668072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.668038 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:53.668072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:53.668066 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7hnn\" (UniqueName: \"kubernetes.io/projected/0b6e68c5-3cd1-4202-a238-e10d85c61a6d-kube-api-access-c7hnn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:54.268779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.268745 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" Nov 02 22:27:54.268779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.268763 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx" event={"ID":"0b6e68c5-3cd1-4202-a238-e10d85c61a6d","Type":"ContainerDied","Data":"b0a2270ab6fc94932bdf40c9cd108fdff3687be93c2f56b3d3140493ba702acf"} Nov 02 22:27:54.271672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.268809 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0a2270ab6fc94932bdf40c9cd108fdff3687be93c2f56b3d3140493ba702acf" Nov 02 22:27:54.429182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.429144 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:54.481103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.481065 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-kube-api-access-4ws5x\" (UniqueName: \"kubernetes.io/projected/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-kube-api-access-4ws5x\") pod \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " Nov 02 22:27:54.481293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.481248 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-openshift-ca\") pod \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " Nov 02 22:27:54.481293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.481275 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-xks-ca\") pod \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\" (UID: \"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c\") " Nov 02 22:27:54.481789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.481761 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-openshift-ca" pod "02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" (UID: "02c4b4e5-562e-49b0-b0d4-7cb4dd76013c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:54.482128 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.482111 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:54.483559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.483530 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-kube-api-access-4ws5x" pod "02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" (UID: "02c4b4e5-562e-49b0-b0d4-7cb4dd76013c"). InnerVolumeSpecName "kube-api-access-4ws5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:54.483559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.483594 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-xks-ca" pod "02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" (UID: "02c4b4e5-562e-49b0-b0d4-7cb4dd76013c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:54.583518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.583433 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ws5x\" (UniqueName: \"kubernetes.io/projected/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-kube-api-access-4ws5x\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:54.583518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:54.583465 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.280771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:55.280740 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" Nov 02 22:27:55.280771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:55.280765 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv" event={"ID":"02c4b4e5-562e-49b0-b0d4-7cb4dd76013c","Type":"ContainerDied","Data":"084b024f77d2a1fa7f25e8acbc8dcc89da957f0c77d52ff1ade416858685b1bd"} Nov 02 22:27:55.281389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:55.280815 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="084b024f77d2a1fa7f25e8acbc8dcc89da957f0c77d52ff1ade416858685b1bd" Nov 02 22:27:56.493135 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:56.493088 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:27:56.914781 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:56.914746 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7216-773e6d"] Nov 02 22:27:56.914995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:56.914942 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7216-773e6d" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" containerName="test-curl-7216-773e6d" containerID="cri-o://86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe" gracePeriod=2 Nov 02 22:27:56.921120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:56.921093 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7216-773e6d"] Nov 02 22:27:56.966493 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:56.966445 5109 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7216-773e6d\" is forbidden: User \"system:node:ip-10-0-143-75.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-75.us-west-2.compute.internal' and this object" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:57.615545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.615508 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7"] Nov 02 22:27:57.616320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616299 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" containerName="test-curl-7216-773e6d" Nov 02 22:27:57.616430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616321 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" containerName="test-curl-7216-773e6d" Nov 02 22:27:57.616430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616353 5109 cpu_manager.go:414] "containerMap: removing container" podUID="02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" containerName="revoke-keys" Nov 02 22:27:57.616430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616361 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" containerName="revoke-keys" Nov 02 22:27:57.616430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616391 5109 cpu_manager.go:414] "containerMap: removing container" podUID="53060a2e-b077-4b08-a8db-704187f0e105" containerName="revoke-keys" Nov 02 22:27:57.616430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616400 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53060a2e-b077-4b08-a8db-704187f0e105" containerName="revoke-keys" Nov 02 22:27:57.616430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616426 5109 cpu_manager.go:414] "containerMap: removing container" podUID="0b6e68c5-3cd1-4202-a238-e10d85c61a6d" containerName="revoke-keys" Nov 02 22:27:57.616430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616433 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6e68c5-3cd1-4202-a238-e10d85c61a6d" containerName="revoke-keys" Nov 02 22:27:57.616749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616645 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="53060a2e-b077-4b08-a8db-704187f0e105" containerName="revoke-keys" Nov 02 22:27:57.616749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616663 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6e68c5-3cd1-4202-a238-e10d85c61a6d" containerName="revoke-keys" Nov 02 22:27:57.616749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616676 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" containerName="test-curl-7216-773e6d" Nov 02 22:27:57.616749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.616685 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" containerName="revoke-keys" Nov 02 22:27:57.621021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.620999 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.731297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.731261 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7"] Nov 02 22:27:57.734120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734093 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734139 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734256 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734219 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztmr\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-kube-api-access-qztmr\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734362 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9c96c-a9f5-4212-aac8-2379505761d7-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734577 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734540 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734642 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734724 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734835 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734819 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.734962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.734881 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.790260 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:57.790219 5109 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7216-773e6d\" is forbidden: User \"system:node:ip-10-0-143-75.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-75.us-west-2.compute.internal' and this object" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:57.836650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.836613 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9c96c-a9f5-4212-aac8-2379505761d7-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.836843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.836750 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.836843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.836782 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.836843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.836829 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.836868 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.836899 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.836982 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.837004 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.837096 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztmr\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-kube-api-access-qztmr\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.837156 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.837257 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.837367 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.837439 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9c96c-a9f5-4212-aac8-2379505761d7-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.837582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.837550 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.839640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.839605 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.839761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.839746 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.898632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.898525 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.898632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.898612 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztmr\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-kube-api-access-qztmr\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:57.937253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:57.937210 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:58.372162 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:27:58.372119 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b9c96c_a9f5_4212_aac8_2379505761d7.slice/crio-bd7f657d4d4edbeaaf4b1d5c753b8bf9d9d92c32b8239e0d27494b2b0280fde3 WatchSource:0}: Error finding container bd7f657d4d4edbeaaf4b1d5c753b8bf9d9d92c32b8239e0d27494b2b0280fde3: Status 404 returned error can't find the container with id bd7f657d4d4edbeaaf4b1d5c753b8bf9d9d92c32b8239e0d27494b2b0280fde3 Nov 02 22:27:58.375008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:58.374976 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:58.375105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:58.375042 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:58.375105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:58.375071 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:27:58.387695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:58.378723 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7"] Nov 02 22:27:59.205056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.202788 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:59.258538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.258498 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53d5200-d456-4a79-b4d8-7c01c0bc3c42-kube-api-access-8shhv\" (UniqueName: \"kubernetes.io/projected/c53d5200-d456-4a79-b4d8-7c01c0bc3c42-kube-api-access-8shhv\") pod \"c53d5200-d456-4a79-b4d8-7c01c0bc3c42\" (UID: \"c53d5200-d456-4a79-b4d8-7c01c0bc3c42\") " Nov 02 22:27:59.261077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.261051 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53d5200-d456-4a79-b4d8-7c01c0bc3c42-kube-api-access-8shhv" pod "c53d5200-d456-4a79-b4d8-7c01c0bc3c42" (UID: "c53d5200-d456-4a79-b4d8-7c01c0bc3c42"). InnerVolumeSpecName "kube-api-access-8shhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:59.319051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.319013 5109 generic.go:356] "Generic (PLEG): container finished" podID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" containerID="86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe" exitCode=137 Nov 02 22:27:59.319051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.319050 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:59.319307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.319103 5109 scope.go:122] "RemoveContainer" containerID="86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe" Nov 02 22:27:59.321391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.321357 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" event={"ID":"a0b9c96c-a9f5-4212-aac8-2379505761d7","Type":"ContainerStarted","Data":"3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272"} Nov 02 22:27:59.321506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.321422 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" event={"ID":"a0b9c96c-a9f5-4212-aac8-2379505761d7","Type":"ContainerStarted","Data":"bd7f657d4d4edbeaaf4b1d5c753b8bf9d9d92c32b8239e0d27494b2b0280fde3"} Nov 02 22:27:59.357535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.357515 5109 scope.go:122] "RemoveContainer" containerID="86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe" Nov 02 22:27:59.357899 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:59.357878 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe\": container with ID starting with 86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe not found: ID does not exist" containerID="86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe" Nov 02 22:27:59.357955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.357907 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe"} err="failed to get container status \"86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe\": rpc error: code = NotFound desc = could not find container \"86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe\": container with ID starting with 86317c9997ae93e41340fd62b466a819b361add80576cc1b1268de66712a35fe not found: ID does not exist" Nov 02 22:27:59.361245 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.361219 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shhv\" (UniqueName: \"kubernetes.io/projected/c53d5200-d456-4a79-b4d8-7c01c0bc3c42-kube-api-access-8shhv\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:59.452418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.452345 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podStartSLOduration=2.452328811 podStartE2EDuration="2.452328811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:59.447985459 +0000 UTC m=+1315.712978678" watchObservedRunningTime="2026-11-02 22:27:59.452328811 +0000 UTC m=+1315.717322031" Nov 02 22:27:59.499659 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:27:59.499531 5109 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7216-773e6d\" is forbidden: User \"system:node:ip-10-0-143-75.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-75.us-west-2.compute.internal' and this object" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" pod="opendatahub/test-curl-7216-773e6d" Nov 02 22:27:59.893302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.893269 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53d5200-d456-4a79-b4d8-7c01c0bc3c42" path="/var/lib/kubelet/pods/c53d5200-d456-4a79-b4d8-7c01c0bc3c42/volumes" Nov 02 22:27:59.946593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.943876 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:27:59.949862 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.949826 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:27:59.950075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:27:59.949887 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:00.948597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:00.946552 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:00.948597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:00.946639 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:01.942720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:01.942679 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:01.942995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:01.942963 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:02.946590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:02.946036 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:02.946590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:02.946101 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:03.978592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:03.973097 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:03.978592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:03.973163 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:04.472479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:04.457100 5109 scope.go:122] "RemoveContainer" containerID="be93fad1c53cdcda9ffde4e65f872249de2e1f3833e1b53647dea52bcfb6ae15" Nov 02 22:28:04.656611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:04.623792 5109 scope.go:122] "RemoveContainer" containerID="4f5142513e1494d976746a844b1c8f67aa1a356a4dc32222275496cf68486484" Nov 02 22:28:04.848518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:04.848083 5109 scope.go:122] "RemoveContainer" containerID="e18234908bae927d828e2d9850801031a89b4366e5a05ce12838da949781f4be" Nov 02 22:28:04.971234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:04.971188 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.098190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:04.989246 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:05.098885 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.098836 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:05.099525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.099493 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.297511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.297462 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.311768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.310611 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:28:05.312909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.312871 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.567597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.559484 5109 scope.go:122] "RemoveContainer" containerID="8e18f7ea67686b625037dc4ecd1c844028e6fa5f1ca8a0ff5dd732247dc06a38" Nov 02 22:28:05.879596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.868733 5109 scope.go:122] "RemoveContainer" containerID="844f7db90ca062f927670a109681b6466c4c16afa5dce02a8484341338ddf74a" Nov 02 22:28:05.879596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.869137 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.968060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.968022 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:05.968354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.968315 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:05.989386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:05.986919 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.150555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.141649 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.342194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.342145 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.345504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.345465 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:06.346135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.346098 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:06.488628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.487867 5109 scope.go:122] "RemoveContainer" containerID="28445922a2717ccc95fe5a3ae30b8ffc2f667a9773f3d76142c06f1a30a35855" Nov 02 22:28:06.491728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.491698 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:28:06.496227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.496205 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:28:06.496897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.496870 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:28:06.497088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.497066 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:28:06.497220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.497209 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:28:06.930617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.905382 5109 scope.go:122] "RemoveContainer" containerID="7d76973a734b2355c4411cecc9d7917afa32df1bb5cd2dcf7741c9c7a81392f8" Nov 02 22:28:06.988029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.946071 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:28:07.021284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.946778 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:07.021284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:06.988793 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:07.128609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.126967 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:28:07.216213 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.216165 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:07.241813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.241762 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:28:07.604234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.604180 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:07.663837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.663781 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:07.729383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.729333 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:07.792289 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.768777 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:07.991901 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:07.991797 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.079596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.079406 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.100318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.080765 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:28:08.315662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.252088 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:08.339771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.339714 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:08.340594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.295831 5109 scope.go:122] "RemoveContainer" containerID="aff18e9d9582585d4bf72848d17d4d1410a2ea6612c5cf9ff60b4a940549b733" Nov 02 22:28:08.402374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.402327 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.592818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.592711 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:28:08.631055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.630987 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.896377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.894288 5109 scope.go:122] "RemoveContainer" containerID="366b6602254511e8ab3d04ce338cc63737f1fe0b0308aeb02ca83f69f2d7e2c8" Nov 02 22:28:08.998592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.992773 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:08.998592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:08.992855 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:09.118554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.118500 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:09.141998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.118842 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:09.249013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.248971 5109 scope.go:122] "RemoveContainer" containerID="e8a244d89e0236df5e8e6053c658ce5ffdaf453d9c02e67e30e46a0c74541107" Nov 02 22:28:09.269875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.269831 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:09.443592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.441377 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:28:09.598930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.598887 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:09.600054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.600028 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:28:09.650814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.650761 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:09.778589 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:09.774294 5109 scope.go:122] "RemoveContainer" containerID="c1a20afc81f4908586bb1c7bf2dff07c5e657436d2803d3eb3aecf6da40fb6bd" Nov 02 22:28:10.134947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.134882 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:10.342802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.171215 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:10.343431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.219739 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:10.343671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.343634 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:10.343814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.295106 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:10.599748 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.599689 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:10.633872 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.623961 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:28:10.661387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.661344 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:10.741410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.741374 5109 scope.go:122] "RemoveContainer" containerID="46f93a88d6017ec1e09fbfd4877e433cba6009fcf4b891b9f5e8ff53031b4682" Nov 02 22:28:10.993782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.993692 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:10.994036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.994004 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:10.994539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:10.994509 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:11.391771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:11.388708 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:11.391771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:11.389140 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:11.457392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:11.457347 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:28:11.615601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:11.612397 5109 scope.go:122] "RemoveContainer" containerID="ec35ec4046b8db999b1514ae0602e0868f7221d3db0be5e839068d561d9c18fb" Nov 02 22:28:11.645120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:11.645014 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:28:11.646118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:11.646085 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:11.672345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:11.663770 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.125093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.116185 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:12.125093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.116260 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:12.279600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.276714 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:12.279600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.276826 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.485598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.404383 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.485598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.425614 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:28:12.485598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.458662 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.666890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.666838 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.667955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.667918 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.974952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.696534 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:28:12.975350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.726745 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.975523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.942126 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:12.975733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.975688 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.975898 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.942224 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:28:12.996358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.996133 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:12.996358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.996208 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:12.997330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.996884 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:12.997330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:12.996934 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:13.425865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.423108 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:13.467636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.426906 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:13.467636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.464036 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:13.467636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.464123 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:13.472041 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.469233 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:13.704410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.674370 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:13.977856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.977761 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:28:14.132816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.124768 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 22:28:14.132816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.055079 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:28:14.132816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:13.978104 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:14.428176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.428117 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:14.428846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.428779 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:28:14.519095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.471682 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:14.519095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.472315 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:14.686625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.674925 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:14.907133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:14.904515 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:28:15.118590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.112559 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:28:15.118590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.114183 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:28:15.220233 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.219973 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:15.220861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.220641 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:15.232331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.232303 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:28:15.232626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.232554 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:28:15.281876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.281847 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:28:15.956018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.955926 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:15.956018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:15.956003 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:16.788996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:16.788960 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:28:16.790553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:16.790529 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:28:17.658079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.658040 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm"] Nov 02 22:28:17.667689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.667659 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp"] Nov 02 22:28:17.667850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.667781 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.672393 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.672354 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.718284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.718248 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm"] Nov 02 22:28:17.726437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.726397 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp"] Nov 02 22:28:17.761220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.761181 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:28:17.761393 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.761250 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:28:17.772323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.772292 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.772521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.772490 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsknc\" (UniqueName: \"kubernetes.io/projected/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-kube-api-access-qsknc\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.772704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.772557 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.772704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.772667 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9hj\" (UniqueName: \"kubernetes.io/projected/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-kube-api-access-nt9hj\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.772816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.772716 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.772900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.772882 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.874106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.874063 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.874338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.874206 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsknc\" (UniqueName: \"kubernetes.io/projected/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-kube-api-access-qsknc\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.874338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.874245 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.874338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.874269 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9hj\" (UniqueName: \"kubernetes.io/projected/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-kube-api-access-nt9hj\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.874338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.874308 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.874595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.874377 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.875100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.874992 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.875100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.875063 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.876955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.876931 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.877066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.876956 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.936229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.936139 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsknc\" (UniqueName: \"kubernetes.io/projected/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-kube-api-access-qsknc\") pod \"maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.936229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.936196 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9hj\" (UniqueName: \"kubernetes.io/projected/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-kube-api-access-nt9hj\") pod \"maas-api-revoke-sub-b81f6b7b-5przp\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:17.987408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.987373 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:17.993553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:17.993530 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:18.382630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:18.378792 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm"] Nov 02 22:28:18.407191 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:28:18.407163 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1775162a_b91b_4fc5_a9e4_332d9f86f9fa.slice/crio-23b835c8cefe09a295075ee2edd3a20d708a259f6b029126981a4ae0ce45ceab WatchSource:0}: Error finding container 23b835c8cefe09a295075ee2edd3a20d708a259f6b029126981a4ae0ce45ceab: Status 404 returned error can't find the container with id 23b835c8cefe09a295075ee2edd3a20d708a259f6b029126981a4ae0ce45ceab Nov 02 22:28:18.422022 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:28:18.421994 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323d2ab7_4eec_4a0c_98fb_18e795f6c77e.slice/crio-117a694798eea849ebaef0fdf0a57e5a6d948ca55e4e57f66c262ed97bb63024 WatchSource:0}: Error finding container 117a694798eea849ebaef0fdf0a57e5a6d948ca55e4e57f66c262ed97bb63024: Status 404 returned error can't find the container with id 117a694798eea849ebaef0fdf0a57e5a6d948ca55e4e57f66c262ed97bb63024 Nov 02 22:28:18.427327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:18.427303 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp"] Nov 02 22:28:19.411640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:19.411606 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" event={"ID":"1775162a-b91b-4fc5-a9e4-332d9f86f9fa","Type":"ContainerStarted","Data":"d75f81e94848f00f604f944392dfc5a14c76b9e9bbbbb163020c01b5bfb6fdb3"} Nov 02 22:28:19.411989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:19.411672 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" event={"ID":"1775162a-b91b-4fc5-a9e4-332d9f86f9fa","Type":"ContainerStarted","Data":"23b835c8cefe09a295075ee2edd3a20d708a259f6b029126981a4ae0ce45ceab"} Nov 02 22:28:19.413391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:19.413368 5109 generic.go:356] "Generic (PLEG): container finished" podID="323d2ab7-4eec-4a0c-98fb-18e795f6c77e" containerID="3522ad772dcc8a1d0ac2472ee880ffaa2091e908095e660ef82086fc29514fa9" exitCode=0 Nov 02 22:28:19.413493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:19.413448 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" event={"ID":"323d2ab7-4eec-4a0c-98fb-18e795f6c77e","Type":"ContainerDied","Data":"3522ad772dcc8a1d0ac2472ee880ffaa2091e908095e660ef82086fc29514fa9"} Nov 02 22:28:19.413537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:19.413491 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" event={"ID":"323d2ab7-4eec-4a0c-98fb-18e795f6c77e","Type":"ContainerStarted","Data":"117a694798eea849ebaef0fdf0a57e5a6d948ca55e4e57f66c262ed97bb63024"} Nov 02 22:28:19.534331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:19.534267 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" podStartSLOduration=2.534245996 podStartE2EDuration="2.534245996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:28:19.528453892 +0000 UTC m=+1335.793447116" watchObservedRunningTime="2026-11-02 22:28:19.534245996 +0000 UTC m=+1335.799239220" Nov 02 22:28:20.424011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.423961 5109 generic.go:356] "Generic (PLEG): container finished" podID="1775162a-b91b-4fc5-a9e4-332d9f86f9fa" containerID="d75f81e94848f00f604f944392dfc5a14c76b9e9bbbbb163020c01b5bfb6fdb3" exitCode=0 Nov 02 22:28:20.424499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.424098 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" event={"ID":"1775162a-b91b-4fc5-a9e4-332d9f86f9fa","Type":"ContainerDied","Data":"d75f81e94848f00f604f944392dfc5a14c76b9e9bbbbb163020c01b5bfb6fdb3"} Nov 02 22:28:20.590174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.590148 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:20.729489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.729401 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-xks-ca\") pod \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " Nov 02 22:28:20.729489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.729457 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-openshift-ca\") pod \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " Nov 02 22:28:20.729758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.729606 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-kube-api-access-nt9hj\" (UniqueName: \"kubernetes.io/projected/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-kube-api-access-nt9hj\") pod \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\" (UID: \"323d2ab7-4eec-4a0c-98fb-18e795f6c77e\") " Nov 02 22:28:20.729896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.729874 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-openshift-ca" pod "323d2ab7-4eec-4a0c-98fb-18e795f6c77e" (UID: "323d2ab7-4eec-4a0c-98fb-18e795f6c77e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:20.730334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.730313 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:20.731904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.731879 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-xks-ca" pod "323d2ab7-4eec-4a0c-98fb-18e795f6c77e" (UID: "323d2ab7-4eec-4a0c-98fb-18e795f6c77e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:20.732000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.731982 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-kube-api-access-nt9hj" pod "323d2ab7-4eec-4a0c-98fb-18e795f6c77e" (UID: "323d2ab7-4eec-4a0c-98fb-18e795f6c77e"). InnerVolumeSpecName "kube-api-access-nt9hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:20.832147 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.832112 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt9hj\" (UniqueName: \"kubernetes.io/projected/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-kube-api-access-nt9hj\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:20.832147 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:20.832144 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d2ab7-4eec-4a0c-98fb-18e795f6c77e-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:21.434276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.434238 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" Nov 02 22:28:21.434797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.434273 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp" event={"ID":"323d2ab7-4eec-4a0c-98fb-18e795f6c77e","Type":"ContainerDied","Data":"117a694798eea849ebaef0fdf0a57e5a6d948ca55e4e57f66c262ed97bb63024"} Nov 02 22:28:21.434797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.434360 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="117a694798eea849ebaef0fdf0a57e5a6d948ca55e4e57f66c262ed97bb63024" Nov 02 22:28:21.612091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.612062 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:21.745888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.745795 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-kube-api-access-qsknc\" (UniqueName: \"kubernetes.io/projected/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-kube-api-access-qsknc\") pod \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " Nov 02 22:28:21.746071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.746025 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-xks-ca\") pod \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " Nov 02 22:28:21.746222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.746202 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-openshift-ca\") pod \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\" (UID: \"1775162a-b91b-4fc5-a9e4-332d9f86f9fa\") " Nov 02 22:28:21.746801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.746666 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-openshift-ca" pod "1775162a-b91b-4fc5-a9e4-332d9f86f9fa" (UID: "1775162a-b91b-4fc5-a9e4-332d9f86f9fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:21.747057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.747035 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:21.748377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.748347 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-xks-ca" pod "1775162a-b91b-4fc5-a9e4-332d9f86f9fa" (UID: "1775162a-b91b-4fc5-a9e4-332d9f86f9fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:21.748662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.748636 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-kube-api-access-qsknc" pod "1775162a-b91b-4fc5-a9e4-332d9f86f9fa" (UID: "1775162a-b91b-4fc5-a9e4-332d9f86f9fa"). InnerVolumeSpecName "kube-api-access-qsknc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:21.849125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.849084 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsknc\" (UniqueName: \"kubernetes.io/projected/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-kube-api-access-qsknc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:21.849125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:21.849119 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1775162a-b91b-4fc5-a9e4-332d9f86f9fa-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:22.481738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:22.481701 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" Nov 02 22:28:22.482225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:22.481938 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm" event={"ID":"1775162a-b91b-4fc5-a9e4-332d9f86f9fa","Type":"ContainerDied","Data":"23b835c8cefe09a295075ee2edd3a20d708a259f6b029126981a4ae0ce45ceab"} Nov 02 22:28:22.482225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:22.481996 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b835c8cefe09a295075ee2edd3a20d708a259f6b029126981a4ae0ce45ceab" Nov 02 22:28:26.497466 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:28:26.497418 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:28:29.213147 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:29.213113 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx"] Nov 02 22:28:29.336896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:29.336847 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww"] Nov 02 22:28:33.382909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.382864 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l"] Nov 02 22:28:33.408701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.408667 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1775162a-b91b-4fc5-a9e4-332d9f86f9fa" containerName="revoke-keys" Nov 02 22:28:33.409722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.409699 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1775162a-b91b-4fc5-a9e4-332d9f86f9fa" containerName="revoke-keys" Nov 02 22:28:33.409963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.409933 5109 cpu_manager.go:414] "containerMap: removing container" podUID="323d2ab7-4eec-4a0c-98fb-18e795f6c77e" containerName="revoke-keys" Nov 02 22:28:33.410058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.410047 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323d2ab7-4eec-4a0c-98fb-18e795f6c77e" containerName="revoke-keys" Nov 02 22:28:33.410441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.410423 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1775162a-b91b-4fc5-a9e4-332d9f86f9fa" containerName="revoke-keys" Nov 02 22:28:33.410601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.410588 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="323d2ab7-4eec-4a0c-98fb-18e795f6c77e" containerName="revoke-keys" Nov 02 22:28:33.474593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.468040 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l"] Nov 02 22:28:33.474593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.468364 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.536639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.529776 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8"] Nov 02 22:28:33.559718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.546011 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.559718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.546145 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chgw\" (UniqueName: \"kubernetes.io/projected/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-kube-api-access-6chgw\") pod \"payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.612469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.612430 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7f1542e5-dockercfg-sf2xs\"" Nov 02 22:28:33.645236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.631788 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7f1542e5\"" Nov 02 22:28:33.645414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.633776 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:33.645963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.644462 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8"] Nov 02 22:28:33.654416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.648829 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.654416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.648906 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-plugins-config-volume\") pod \"payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:33.654416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.648988 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chgw\" (UniqueName: \"kubernetes.io/projected/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-kube-api-access-6chgw\") pod \"payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.654416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.649043 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkllh\" (UniqueName: \"kubernetes.io/projected/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-kube-api-access-kkllh\") pod \"payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:33.680046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.679997 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.717359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.717272 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chgw\" (UniqueName: \"kubernetes.io/projected/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-kube-api-access-6chgw\") pod \"payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.740252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.739997 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:33.755599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.755396 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-plugins-config-volume\") pod \"payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:33.755599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.755515 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkllh\" (UniqueName: \"kubernetes.io/projected/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-kube-api-access-kkllh\") pod \"payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:33.804373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.757486 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-plugins-config-volume\") pod \"payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:33.824597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:33.824552 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkllh\" (UniqueName: \"kubernetes.io/projected/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-kube-api-access-kkllh\") pod \"payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:34.124351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.124317 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:34.213695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.213645 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" podUID="e3b3e669-f53f-4070-bbc7-408628639dca" containerName="payload-processing" containerID="cri-o://cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10" gracePeriod=25 Nov 02 22:28:34.337300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.337248 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" podUID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" containerName="payload-pre-processing" containerID="cri-o://ee376daaa0bbdeb912e64220da5f734e7617ccc730d5173f6864c8166ae3c23b" gracePeriod=25 Nov 02 22:28:34.387781 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:28:34.387747 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98cd2ba_cd76_45b7_ab54_2725562ed2bd.slice/crio-3e75e1d7ff24c36092956a73cd7dc404cc5d12339f0424e87a206c6fc6014b92 WatchSource:0}: Error finding container 3e75e1d7ff24c36092956a73cd7dc404cc5d12339f0424e87a206c6fc6014b92: Status 404 returned error can't find the container with id 3e75e1d7ff24c36092956a73cd7dc404cc5d12339f0424e87a206c6fc6014b92 Nov 02 22:28:34.393090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.393061 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l"] Nov 02 22:28:34.507000 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:28:34.506961 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf545a4de_d1b6_4304_8fa4_efb9dab2bc86.slice/crio-731118d297f7dbf3e01c55200f9d9eed077475d7befaadd813e24aaa0c83c7a6 WatchSource:0}: Error finding container 731118d297f7dbf3e01c55200f9d9eed077475d7befaadd813e24aaa0c83c7a6: Status 404 returned error can't find the container with id 731118d297f7dbf3e01c55200f9d9eed077475d7befaadd813e24aaa0c83c7a6 Nov 02 22:28:34.515588 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.515540 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8"] Nov 02 22:28:34.655674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.655624 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" event={"ID":"a98cd2ba-cd76-45b7-ab54-2725562ed2bd","Type":"ContainerStarted","Data":"3a4f5510adac619e24e1fb38cf129b8a8eb160091c1a66ffbb3d790c70df61bf"} Nov 02 22:28:34.655864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.655708 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:34.655864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.655734 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" event={"ID":"a98cd2ba-cd76-45b7-ab54-2725562ed2bd","Type":"ContainerStarted","Data":"3e75e1d7ff24c36092956a73cd7dc404cc5d12339f0424e87a206c6fc6014b92"} Nov 02 22:28:34.657463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.657429 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" event={"ID":"f545a4de-d1b6-4304-8fa4-efb9dab2bc86","Type":"ContainerStarted","Data":"a555e11a2e32cc0de316141fc0471832cccfad109335e4d7e8eec2d81a9e13e7"} Nov 02 22:28:34.657614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.657492 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" event={"ID":"f545a4de-d1b6-4304-8fa4-efb9dab2bc86","Type":"ContainerStarted","Data":"731118d297f7dbf3e01c55200f9d9eed077475d7befaadd813e24aaa0c83c7a6"} Nov 02 22:28:34.657614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.657520 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:34.774549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.774481 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" podStartSLOduration=1.774461052 podStartE2EDuration="1.774461052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:28:34.771701425 +0000 UTC m=+1351.036694646" watchObservedRunningTime="2026-11-02 22:28:34.774461052 +0000 UTC m=+1351.039454272" Nov 02 22:28:34.950590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:34.946970 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" podStartSLOduration=1.946944035 podStartE2EDuration="1.946944035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:28:34.912144502 +0000 UTC m=+1351.177137723" watchObservedRunningTime="2026-11-02 22:28:34.946944035 +0000 UTC m=+1351.211937255" Nov 02 22:28:35.019592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:35.015548 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" podUID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:28:35.995851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:35.995812 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" podUID="e3b3e669-f53f-4070-bbc7-408628639dca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:28:38.396506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:38.393686 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:38.428788 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:38.428736 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:38.925481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:38.924465 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:39.394746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:39.394695 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Nov 02 22:28:39.395453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:39.395428 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:28:39.684910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:39.684805 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:39.918353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:39.917729 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:39.933283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:39.933237 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:39.934109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:39.934084 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:28:40.029752 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.029684 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.193594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.188738 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:40.193594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.188870 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.419651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.418939 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.419651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.419532 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.714678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.714502 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:40.717219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.717179 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:28:40.768661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.767109 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Nov 02 22:28:40.844289 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.844225 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.924417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.924375 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:40.935027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.933938 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:28:40.948609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:40.935743 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:28:41.169516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.169474 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:28:41.230172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.230136 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:28:41.474884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.434399 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:28:41.474884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.435548 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:28:41.496279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.484167 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:41.764659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.727936 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:41.984595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.950664 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:28:41.984595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:41.951811 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:42.231008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.230904 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:42.543596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.493341 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:42.543596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.455988 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 22:28:42.712398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.712349 5109 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 02 22:28:42.787593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.776368 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:42.927483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.927291 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:42.927798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.927763 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:42.951755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.951706 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 22:28:42.953505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:42.953029 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 22:28:43.240176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.239701 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:28:43.240176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.239774 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:43.242780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.242219 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:43.567595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.544243 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:28:43.567595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.546150 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:43.884676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.880751 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:28:43.904627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.886676 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 22:28:43.975680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:43.974662 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:44.022585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:44.022534 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:28:44.026594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:44.025482 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:28:44.044011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:44.043957 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:28:44.045096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:44.045071 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:28:44.998668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:44.998395 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" podUID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:28:45.733500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:45.733468 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:28:45.777971 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:45.777936 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:28:45.996366 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:45.996272 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" podUID="e3b3e669-f53f-4070-bbc7-408628639dca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:28:54.610210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.608752 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:28:54.691446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.690749 5109 generic.go:356] "Generic (PLEG): container finished" podID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" containerID="ee376daaa0bbdeb912e64220da5f734e7617ccc730d5173f6864c8166ae3c23b" exitCode=0 Nov 02 22:28:54.691446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.690962 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" event={"ID":"6a6cc8e5-2298-4660-8b87-d9c23fcdccac","Type":"ContainerDied","Data":"ee376daaa0bbdeb912e64220da5f734e7617ccc730d5173f6864c8166ae3c23b"} Nov 02 22:28:54.694634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.693539 5109 generic.go:356] "Generic (PLEG): container finished" podID="e3b3e669-f53f-4070-bbc7-408628639dca" containerID="cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10" exitCode=0 Nov 02 22:28:54.694634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.693807 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" event={"ID":"e3b3e669-f53f-4070-bbc7-408628639dca","Type":"ContainerDied","Data":"cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10"} Nov 02 22:28:54.694634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.693865 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" event={"ID":"e3b3e669-f53f-4070-bbc7-408628639dca","Type":"ContainerDied","Data":"7f58c7bfce8aae581580c68c8a22793dc4e9a7f59be530b895c045eaca1f78c0"} Nov 02 22:28:54.694634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.693900 5109 scope.go:122] "RemoveContainer" containerID="cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10" Nov 02 22:28:54.694634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.694084 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx" Nov 02 22:28:54.712739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.712644 5109 scope.go:122] "RemoveContainer" containerID="cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10" Nov 02 22:28:54.713433 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:28:54.713362 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10\": container with ID starting with cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10 not found: ID does not exist" containerID="cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10" Nov 02 22:28:54.713433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.713397 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10"} err="failed to get container status \"cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10\": rpc error: code = NotFound desc = could not find container \"cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10\": container with ID starting with cad1e67ab246b7edec902884a43dd61a8fdcdccd32bdb052a887ac2be82c9b10 not found: ID does not exist" Nov 02 22:28:54.773630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.773599 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b3e669-f53f-4070-bbc7-408628639dca-kube-api-access-gvkvs\" (UniqueName: \"kubernetes.io/projected/e3b3e669-f53f-4070-bbc7-408628639dca-kube-api-access-gvkvs\") pod \"e3b3e669-f53f-4070-bbc7-408628639dca\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " Nov 02 22:28:54.774008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.773982 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b3e669-f53f-4070-bbc7-408628639dca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b3e669-f53f-4070-bbc7-408628639dca-plugins-config-volume\") pod \"e3b3e669-f53f-4070-bbc7-408628639dca\" (UID: \"e3b3e669-f53f-4070-bbc7-408628639dca\") " Nov 02 22:28:54.775085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.775052 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3e669-f53f-4070-bbc7-408628639dca-plugins-config-volume" pod "e3b3e669-f53f-4070-bbc7-408628639dca" (UID: "e3b3e669-f53f-4070-bbc7-408628639dca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:54.776288 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.776264 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b3e669-f53f-4070-bbc7-408628639dca-kube-api-access-gvkvs" pod "e3b3e669-f53f-4070-bbc7-408628639dca" (UID: "e3b3e669-f53f-4070-bbc7-408628639dca"). InnerVolumeSpecName "kube-api-access-gvkvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:54.797548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.797516 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:28:54.875946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.875915 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-plugins-config-volume\") pod \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " Nov 02 22:28:54.876232 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.876218 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-kube-api-access-clhzj\" (UniqueName: \"kubernetes.io/projected/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-kube-api-access-clhzj\") pod \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\" (UID: \"6a6cc8e5-2298-4660-8b87-d9c23fcdccac\") " Nov 02 22:28:54.876413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.876385 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-plugins-config-volume" pod "6a6cc8e5-2298-4660-8b87-d9c23fcdccac" (UID: "6a6cc8e5-2298-4660-8b87-d9c23fcdccac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:54.876852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.876834 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkvs\" (UniqueName: \"kubernetes.io/projected/e3b3e669-f53f-4070-bbc7-408628639dca-kube-api-access-gvkvs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:54.876890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.876859 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:54.876890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.876875 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b3e669-f53f-4070-bbc7-408628639dca-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:54.878475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.878451 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-kube-api-access-clhzj" pod "6a6cc8e5-2298-4660-8b87-d9c23fcdccac" (UID: "6a6cc8e5-2298-4660-8b87-d9c23fcdccac"). InnerVolumeSpecName "kube-api-access-clhzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:54.978894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:54.978857 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clhzj\" (UniqueName: \"kubernetes.io/projected/6a6cc8e5-2298-4660-8b87-d9c23fcdccac-kube-api-access-clhzj\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:55.119298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.119209 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx"] Nov 02 22:28:55.124873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.124845 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-711bd4-f99f9cdcc-jptpx"] Nov 02 22:28:55.709884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.709821 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" event={"ID":"6a6cc8e5-2298-4660-8b87-d9c23fcdccac","Type":"ContainerDied","Data":"4c82371bc6ba251e3facdeef8a8742ed37235be408813d7f3d56776e43774a77"} Nov 02 22:28:55.710382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.709950 5109 scope.go:122] "RemoveContainer" containerID="ee376daaa0bbdeb912e64220da5f734e7617ccc730d5173f6864c8166ae3c23b" Nov 02 22:28:55.710382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.710156 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww" Nov 02 22:28:55.879902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.879867 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww"] Nov 02 22:28:55.895801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.895768 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b3e669-f53f-4070-bbc7-408628639dca" path="/var/lib/kubelet/pods/e3b3e669-f53f-4070-bbc7-408628639dca/volumes" Nov 02 22:28:55.929362 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:55.929326 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-711bd4-78d498ff79-jd2ww"] Nov 02 22:28:56.492390 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:28:56.492353 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:28:57.898354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:57.898323 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" path="/var/lib/kubelet/pods/6a6cc8e5-2298-4660-8b87-d9c23fcdccac/volumes" Nov 02 22:28:59.025909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.025879 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r"] Nov 02 22:28:59.026751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.026734 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e3b3e669-f53f-4070-bbc7-408628639dca" containerName="payload-processing" Nov 02 22:28:59.026812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.026753 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b3e669-f53f-4070-bbc7-408628639dca" containerName="payload-processing" Nov 02 22:28:59.026812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.026770 5109 cpu_manager.go:414] "containerMap: removing container" podUID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" containerName="payload-pre-processing" Nov 02 22:28:59.026812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.026777 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" containerName="payload-pre-processing" Nov 02 22:28:59.026950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.026940 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b3e669-f53f-4070-bbc7-408628639dca" containerName="payload-processing" Nov 02 22:28:59.026986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.026954 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6cc8e5-2298-4660-8b87-d9c23fcdccac" containerName="payload-pre-processing" Nov 02 22:28:59.043980 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.043949 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59"] Nov 02 22:28:59.044189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.044065 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.048919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.048895 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.057450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.057414 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dw2\" (UniqueName: \"kubernetes.io/projected/1bcd630a-0ac6-4502-9e59-13076d714458-kube-api-access-r7dw2\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.057604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.057533 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.057679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.057661 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.097121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.097086 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:28:59.097306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.097175 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:28:59.152826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.152789 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59"] Nov 02 22:28:59.159511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.158672 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r"] Nov 02 22:28:59.159511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.159403 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zb4q\" (UniqueName: \"kubernetes.io/projected/1632a11f-1152-406e-8201-98615272f92f-kube-api-access-4zb4q\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.159511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.159484 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1632a11f-1152-406e-8201-98615272f92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.159877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.159748 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1632a11f-1152-406e-8201-98615272f92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.159877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.159788 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dw2\" (UniqueName: \"kubernetes.io/projected/1bcd630a-0ac6-4502-9e59-13076d714458-kube-api-access-r7dw2\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.159877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.159849 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.160044 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.159894 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.161407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.161379 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.163034 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.162995 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.221863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.221834 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dw2\" (UniqueName: \"kubernetes.io/projected/1bcd630a-0ac6-4502-9e59-13076d714458-kube-api-access-r7dw2\") pod \"maas-api-revoke-sub-445bfe47-mkv5r\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.261641 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.261613 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zb4q\" (UniqueName: \"kubernetes.io/projected/1632a11f-1152-406e-8201-98615272f92f-kube-api-access-4zb4q\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.261807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.261719 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1632a11f-1152-406e-8201-98615272f92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.261946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.261929 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1632a11f-1152-406e-8201-98615272f92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.262499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.262477 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1632a11f-1152-406e-8201-98615272f92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.264196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.264174 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1632a11f-1152-406e-8201-98615272f92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.323299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.323221 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zb4q\" (UniqueName: \"kubernetes.io/projected/1632a11f-1152-406e-8201-98615272f92f-kube-api-access-4zb4q\") pod \"maas-api-revoke-sub-72b6680a-l8n59\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.366025 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.365987 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:28:59.375668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.375646 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:28:59.771392 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:28:59.771360 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcd630a_0ac6_4502_9e59_13076d714458.slice/crio-ee8cff7c05f3d4c0be1c496cc83aeb6fe16af70f0d7b57bae39ca29ab87364b9 WatchSource:0}: Error finding container ee8cff7c05f3d4c0be1c496cc83aeb6fe16af70f0d7b57bae39ca29ab87364b9: Status 404 returned error can't find the container with id ee8cff7c05f3d4c0be1c496cc83aeb6fe16af70f0d7b57bae39ca29ab87364b9 Nov 02 22:28:59.776849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.776823 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r"] Nov 02 22:28:59.805528 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.805500 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" event={"ID":"1bcd630a-0ac6-4502-9e59-13076d714458","Type":"ContainerStarted","Data":"ee8cff7c05f3d4c0be1c496cc83aeb6fe16af70f0d7b57bae39ca29ab87364b9"} Nov 02 22:28:59.817393 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:28:59.817351 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1632a11f_1152_406e_8201_98615272f92f.slice/crio-a7d886f7c07d324e8848d495985fd64f85062002bc82525f4f84f6aa8b9e8582 WatchSource:0}: Error finding container a7d886f7c07d324e8848d495985fd64f85062002bc82525f4f84f6aa8b9e8582: Status 404 returned error can't find the container with id a7d886f7c07d324e8848d495985fd64f85062002bc82525f4f84f6aa8b9e8582 Nov 02 22:28:59.823520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:28:59.823495 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59"] Nov 02 22:29:00.744265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.744189 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7"] Nov 02 22:29:00.749104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.749084 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.793977 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794008 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dea2654-d13d-484f-ad7a-982fea3bffec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794053 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794195 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794311 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7fd\" (UniqueName: \"kubernetes.io/projected/7dea2654-d13d-484f-ad7a-982fea3bffec-kube-api-access-vs7fd\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794379 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794487 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794552 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.794659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.794645 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.816354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.816323 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" event={"ID":"1bcd630a-0ac6-4502-9e59-13076d714458","Type":"ContainerStarted","Data":"09708276ee637b40bb9fec61f475d3a6e214038b7114bbf148ef5641b35f84ce"} Nov 02 22:29:00.818164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.818138 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" event={"ID":"1632a11f-1152-406e-8201-98615272f92f","Type":"ContainerStarted","Data":"0b474e647ebe44c0e3bbeb0905d7931ee4c96be1040c9c8ba3b5c73b4ec15b69"} Nov 02 22:29:00.818302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.818184 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" event={"ID":"1632a11f-1152-406e-8201-98615272f92f","Type":"ContainerStarted","Data":"a7d886f7c07d324e8848d495985fd64f85062002bc82525f4f84f6aa8b9e8582"} Nov 02 22:29:00.847819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.847785 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7"] Nov 02 22:29:00.896326 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896289 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.896578 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896539 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.896676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896619 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dea2654-d13d-484f-ad7a-982fea3bffec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.896743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896689 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.896806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896743 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.896806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896766 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.896907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896841 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7fd\" (UniqueName: \"kubernetes.io/projected/7dea2654-d13d-484f-ad7a-982fea3bffec-kube-api-access-vs7fd\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.896907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.896881 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.897043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.897019 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.897103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.897029 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.897161 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.897145 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.897306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.897287 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.897516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.897490 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dea2654-d13d-484f-ad7a-982fea3bffec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.897644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.897494 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.899614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.899588 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.900415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.900392 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.958477 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.958438 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dea2654-d13d-484f-ad7a-982fea3bffec-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:00.958663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:00.958523 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7fd\" (UniqueName: \"kubernetes.io/projected/7dea2654-d13d-484f-ad7a-982fea3bffec-kube-api-access-vs7fd\") pod \"maas-default-gateway-openshift-default-76f4876667-bdbv7\" (UID: \"7dea2654-d13d-484f-ad7a-982fea3bffec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:01.033343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.033283 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" podStartSLOduration=3.033266982 podStartE2EDuration="3.033266982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:00.98482821 +0000 UTC m=+1377.249821430" watchObservedRunningTime="2026-11-02 22:29:01.033266982 +0000 UTC m=+1377.298260242" Nov 02 22:29:01.064866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.064830 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:01.108638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.108556 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" podStartSLOduration=3.108536865 podStartE2EDuration="3.108536865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:01.100720876 +0000 UTC m=+1377.365714095" watchObservedRunningTime="2026-11-02 22:29:01.108536865 +0000 UTC m=+1377.373530110" Nov 02 22:29:01.503054 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:29:01.503008 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dea2654_d13d_484f_ad7a_982fea3bffec.slice/crio-f3872fb5f33ef8432c9b53a1db647c60bccc8cc2c8d847f2f6bdc2ed9cdd75cc WatchSource:0}: Error finding container f3872fb5f33ef8432c9b53a1db647c60bccc8cc2c8d847f2f6bdc2ed9cdd75cc: Status 404 returned error can't find the container with id f3872fb5f33ef8432c9b53a1db647c60bccc8cc2c8d847f2f6bdc2ed9cdd75cc Nov 02 22:29:01.505825 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.505788 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:01.505926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.505877 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:01.505926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.505916 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:01.506993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.506966 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7"] Nov 02 22:29:01.830624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.830589 5109 generic.go:356] "Generic (PLEG): container finished" podID="1632a11f-1152-406e-8201-98615272f92f" containerID="0b474e647ebe44c0e3bbeb0905d7931ee4c96be1040c9c8ba3b5c73b4ec15b69" exitCode=0 Nov 02 22:29:01.831160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.830637 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" event={"ID":"1632a11f-1152-406e-8201-98615272f92f","Type":"ContainerDied","Data":"0b474e647ebe44c0e3bbeb0905d7931ee4c96be1040c9c8ba3b5c73b4ec15b69"} Nov 02 22:29:01.833049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.833014 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" event={"ID":"7dea2654-d13d-484f-ad7a-982fea3bffec","Type":"ContainerStarted","Data":"8cd2827d6e026a204d6dd4ae98f89cc147191dd378f729b239c626cf98ccef7c"} Nov 02 22:29:01.833198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.833081 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" event={"ID":"7dea2654-d13d-484f-ad7a-982fea3bffec","Type":"ContainerStarted","Data":"f3872fb5f33ef8432c9b53a1db647c60bccc8cc2c8d847f2f6bdc2ed9cdd75cc"} Nov 02 22:29:01.834988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.834956 5109 generic.go:356] "Generic (PLEG): container finished" podID="1bcd630a-0ac6-4502-9e59-13076d714458" containerID="09708276ee637b40bb9fec61f475d3a6e214038b7114bbf148ef5641b35f84ce" exitCode=0 Nov 02 22:29:01.835114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:01.835035 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" event={"ID":"1bcd630a-0ac6-4502-9e59-13076d714458","Type":"ContainerDied","Data":"09708276ee637b40bb9fec61f475d3a6e214038b7114bbf148ef5641b35f84ce"} Nov 02 22:29:02.065380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:02.065340 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:02.067313 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:02.067274 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:02.067476 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:02.067334 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:02.071749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:02.071696 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podStartSLOduration=2.071679476 podStartE2EDuration="2.071679476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:02.067912838 +0000 UTC m=+1378.332906059" watchObservedRunningTime="2026-11-02 22:29:02.071679476 +0000 UTC m=+1378.336672694" Nov 02 22:29:03.065814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.065780 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:03.066282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.065878 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:03.163820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.163745 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:29:03.165893 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.165874 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:29:03.258841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.258801 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcd630a-0ac6-4502-9e59-13076d714458-kube-api-access-r7dw2\" (UniqueName: \"kubernetes.io/projected/1bcd630a-0ac6-4502-9e59-13076d714458-kube-api-access-r7dw2\") pod \"1bcd630a-0ac6-4502-9e59-13076d714458\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " Nov 02 22:29:03.259065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.258868 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-openshift-ca\") pod \"1bcd630a-0ac6-4502-9e59-13076d714458\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " Nov 02 22:29:03.259065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.258903 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-xks-ca\") pod \"1bcd630a-0ac6-4502-9e59-13076d714458\" (UID: \"1bcd630a-0ac6-4502-9e59-13076d714458\") " Nov 02 22:29:03.259065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.259006 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1632a11f-1152-406e-8201-98615272f92f-kube-api-access-4zb4q\" (UniqueName: \"kubernetes.io/projected/1632a11f-1152-406e-8201-98615272f92f-kube-api-access-4zb4q\") pod \"1632a11f-1152-406e-8201-98615272f92f\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " Nov 02 22:29:03.259257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.259112 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1632a11f-1152-406e-8201-98615272f92f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1632a11f-1152-406e-8201-98615272f92f-maas-api-openshift-ca\") pod \"1632a11f-1152-406e-8201-98615272f92f\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " Nov 02 22:29:03.259257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.259186 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1632a11f-1152-406e-8201-98615272f92f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1632a11f-1152-406e-8201-98615272f92f-maas-api-xks-ca\") pod \"1632a11f-1152-406e-8201-98615272f92f\" (UID: \"1632a11f-1152-406e-8201-98615272f92f\") " Nov 02 22:29:03.259434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.259308 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-openshift-ca" pod "1bcd630a-0ac6-4502-9e59-13076d714458" (UID: "1bcd630a-0ac6-4502-9e59-13076d714458"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:03.260297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.259501 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1632a11f-1152-406e-8201-98615272f92f-maas-api-openshift-ca" pod "1632a11f-1152-406e-8201-98615272f92f" (UID: "1632a11f-1152-406e-8201-98615272f92f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:03.260297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.260239 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:03.260297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.260268 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1632a11f-1152-406e-8201-98615272f92f-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:03.261483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.261448 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-xks-ca" pod "1bcd630a-0ac6-4502-9e59-13076d714458" (UID: "1bcd630a-0ac6-4502-9e59-13076d714458"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:03.262746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.262721 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1632a11f-1152-406e-8201-98615272f92f-maas-api-xks-ca" pod "1632a11f-1152-406e-8201-98615272f92f" (UID: "1632a11f-1152-406e-8201-98615272f92f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:03.262838 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.262796 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1632a11f-1152-406e-8201-98615272f92f-kube-api-access-4zb4q" pod "1632a11f-1152-406e-8201-98615272f92f" (UID: "1632a11f-1152-406e-8201-98615272f92f"). InnerVolumeSpecName "kube-api-access-4zb4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:03.262908 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.262861 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcd630a-0ac6-4502-9e59-13076d714458-kube-api-access-r7dw2" pod "1bcd630a-0ac6-4502-9e59-13076d714458" (UID: "1bcd630a-0ac6-4502-9e59-13076d714458"). InnerVolumeSpecName "kube-api-access-r7dw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:03.362593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.362453 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dw2\" (UniqueName: \"kubernetes.io/projected/1bcd630a-0ac6-4502-9e59-13076d714458-kube-api-access-r7dw2\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:03.362593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.362500 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcd630a-0ac6-4502-9e59-13076d714458-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:03.362593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.362517 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zb4q\" (UniqueName: \"kubernetes.io/projected/1632a11f-1152-406e-8201-98615272f92f-kube-api-access-4zb4q\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:03.362593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.362532 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1632a11f-1152-406e-8201-98615272f92f-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:03.866206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.866170 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" Nov 02 22:29:03.866206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.866194 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59" event={"ID":"1632a11f-1152-406e-8201-98615272f92f","Type":"ContainerDied","Data":"a7d886f7c07d324e8848d495985fd64f85062002bc82525f4f84f6aa8b9e8582"} Nov 02 22:29:03.866485 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.866244 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7d886f7c07d324e8848d495985fd64f85062002bc82525f4f84f6aa8b9e8582" Nov 02 22:29:03.868410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.868386 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" Nov 02 22:29:03.868561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.868421 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r" event={"ID":"1bcd630a-0ac6-4502-9e59-13076d714458","Type":"ContainerDied","Data":"ee8cff7c05f3d4c0be1c496cc83aeb6fe16af70f0d7b57bae39ca29ab87364b9"} Nov 02 22:29:03.868561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:03.868458 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee8cff7c05f3d4c0be1c496cc83aeb6fe16af70f0d7b57bae39ca29ab87364b9" Nov 02 22:29:04.065655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:04.065619 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:04.065816 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:04.065681 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:05.075594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:05.070708 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:05.075594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:05.070793 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:06.076635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:06.071889 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:06.076635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:06.071950 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:07.067028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:07.066988 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:07.067319 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:07.067292 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:08.069022 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:08.068974 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:08.069529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:08.069043 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:09.065550 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:09.065514 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:09.065895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:09.065844 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:10.065608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:10.065550 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:29:10.065995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:10.065635 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 22:29:11.839592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.832024 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc"] Nov 02 22:29:11.839592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.833328 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1632a11f-1152-406e-8201-98615272f92f" containerName="revoke-keys" Nov 02 22:29:11.839592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.833356 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1632a11f-1152-406e-8201-98615272f92f" containerName="revoke-keys" Nov 02 22:29:11.839592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.833405 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1bcd630a-0ac6-4502-9e59-13076d714458" containerName="revoke-keys" Nov 02 22:29:11.839592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.833414 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcd630a-0ac6-4502-9e59-13076d714458" containerName="revoke-keys" Nov 02 22:29:11.839592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.833635 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcd630a-0ac6-4502-9e59-13076d714458" containerName="revoke-keys" Nov 02 22:29:11.839592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.833665 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1632a11f-1152-406e-8201-98615272f92f" containerName="revoke-keys" Nov 02 22:29:11.920592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:11.914402 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.031595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.023307 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c145e338-1ae0-4d01-889d-f2f48ae7ef16-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.081591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.080236 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbw8\" (UniqueName: \"kubernetes.io/projected/c145e338-1ae0-4d01-889d-f2f48ae7ef16-kube-api-access-cfbw8\") pod \"payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.081591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.033933 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2e86dfb1\"" Nov 02 22:29:12.081591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.076647 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc"] Nov 02 22:29:12.081591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.033768 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2e86dfb1-dockercfg-b62qf\"" Nov 02 22:29:12.121852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.121050 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:12.121852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.121128 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:12.322215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.322165 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c145e338-1ae0-4d01-889d-f2f48ae7ef16-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.323665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.323635 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbw8\" (UniqueName: \"kubernetes.io/projected/c145e338-1ae0-4d01-889d-f2f48ae7ef16-kube-api-access-cfbw8\") pod \"payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.346320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.323949 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c145e338-1ae0-4d01-889d-f2f48ae7ef16-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.409658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.409557 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbw8\" (UniqueName: \"kubernetes.io/projected/c145e338-1ae0-4d01-889d-f2f48ae7ef16-kube-api-access-cfbw8\") pod \"payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.584783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.584744 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:12.831803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.831358 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:12.832005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.831981 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:12.832985 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:12.832966 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" Nov 02 22:29:13.055977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:13.055361 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc"] Nov 02 22:29:13.098009 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:29:13.097963 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc145e338_1ae0_4d01_889d_f2f48ae7ef16.slice/crio-c0cd24a4e58c395b127dd8717f5d168efe918fbf1459d10ab57e5059fde534e0 WatchSource:0}: Error finding container c0cd24a4e58c395b127dd8717f5d168efe918fbf1459d10ab57e5059fde534e0: Status 404 returned error can't find the container with id c0cd24a4e58c395b127dd8717f5d168efe918fbf1459d10ab57e5059fde534e0 Nov 02 22:29:13.494084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:13.493969 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" event={"ID":"c145e338-1ae0-4d01-889d-f2f48ae7ef16","Type":"ContainerStarted","Data":"96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c"} Nov 02 22:29:13.494084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:13.494041 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" event={"ID":"c145e338-1ae0-4d01-889d-f2f48ae7ef16","Type":"ContainerStarted","Data":"c0cd24a4e58c395b127dd8717f5d168efe918fbf1459d10ab57e5059fde534e0"} Nov 02 22:29:13.494084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:13.494071 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:13.611629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:13.611526 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" podStartSLOduration=2.611505939 podStartE2EDuration="2.611505939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:13.61095754 +0000 UTC m=+1389.875950761" watchObservedRunningTime="2026-11-02 22:29:13.611505939 +0000 UTC m=+1389.876499159" Nov 02 22:29:15.147830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.147777 5109 patch_prober.go:29] interesting pod/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:15.148401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.147863 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:15.524889 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.524856 5109 scope.go:122] "RemoveContainer" containerID="6d96984a08941be52b99e555190f05c0888be1792c67ab61b6862b384488eea1" Nov 02 22:29:15.768537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.768501 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6"] Nov 02 22:29:15.775281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.775202 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874046 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874128 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874160 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874214 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874260 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874357 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62nmj\" (UniqueName: \"kubernetes.io/projected/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-kube-api-access-62nmj\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874414 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874552 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:15.893262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:15.874598 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.017454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.017422 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.017749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.017729 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.018072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.018049 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.018223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.018207 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.018348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.018332 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.018514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.018496 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.018696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.018679 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.018801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.018782 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.018888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.018858 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.019083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.019063 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62nmj\" (UniqueName: \"kubernetes.io/projected/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-kube-api-access-62nmj\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.019246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.019229 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.019355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.019069 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.019685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.019660 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.019755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.019146 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.022904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.022879 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.024929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.024905 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.026499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.026439 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6"] Nov 02 22:29:16.079464 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.079432 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.080396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.080363 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62nmj\" (UniqueName: \"kubernetes.io/projected/c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa-kube-api-access-62nmj\") pod \"maas-default-gateway-openshift-default-76f4876667-tk2q6\" (UID: \"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.148883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.148846 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:16.591232 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.591194 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6"] Nov 02 22:29:16.615582 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:29:16.615533 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d8c50f_fd0e_46e1_8fb0_18e5fa0e51fa.slice/crio-0ccfd5d3dca956d46512e4f8920fa471493c2945b09538ecaa0f719bc72e38be WatchSource:0}: Error finding container 0ccfd5d3dca956d46512e4f8920fa471493c2945b09538ecaa0f719bc72e38be: Status 404 returned error can't find the container with id 0ccfd5d3dca956d46512e4f8920fa471493c2945b09538ecaa0f719bc72e38be Nov 02 22:29:16.618619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.618546 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:16.618712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.618655 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:16.618712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:16.618685 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:17.551653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:17.544250 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" event={"ID":"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa","Type":"ContainerStarted","Data":"d5a1b58e8149e86157e2368ea4979bb1623eb4b1b047358ef5d747191ad46c5c"} Nov 02 22:29:17.551653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:17.544328 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" event={"ID":"c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa","Type":"ContainerStarted","Data":"0ccfd5d3dca956d46512e4f8920fa471493c2945b09538ecaa0f719bc72e38be"} Nov 02 22:29:17.750660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:17.748318 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podStartSLOduration=2.748292417 podStartE2EDuration="2.748292417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:17.722344207 +0000 UTC m=+1393.987337425" watchObservedRunningTime="2026-11-02 22:29:17.748292417 +0000 UTC m=+1394.013285637" Nov 02 22:29:18.150008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.149962 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:18.151954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.151926 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:29:18.152039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.151983 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:29:18.560745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.560711 5109 generic.go:356] "Generic (PLEG): container finished" podID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerID="473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570" exitCode=0 Nov 02 22:29:18.561269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.560793 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" event={"ID":"3fa74d8b-b126-487a-8ce9-84724a64e4b2","Type":"ContainerDied","Data":"473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570"} Nov 02 22:29:18.561411 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.561390 5109 scope.go:122] "RemoveContainer" containerID="473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570" Nov 02 22:29:18.563063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.563036 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:18.563129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.563107 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:18.563166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:18.563139 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:29:19.151590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:19.150973 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:29:19.151590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:19.151042 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:29:19.571168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:19.571130 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" event={"ID":"3fa74d8b-b126-487a-8ce9-84724a64e4b2","Type":"ContainerStarted","Data":"f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04"} Nov 02 22:29:19.850654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:19.847542 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:29:19.850654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:19.847697 5109 patch_prober.go:29] interesting pod/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 22:29:19.850654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:19.847744 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 22:29:20.150653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:20.150013 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:29:20.150653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:20.150078 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:29:20.858594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:20.858488 5109 patch_prober.go:29] interesting pod/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 22:29:20.858594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:20.858552 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 22:29:21.169702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:21.163961 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:29:21.169702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:21.164024 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:29:21.869595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:21.866439 5109 patch_prober.go:29] interesting pod/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 22:29:21.869595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:21.866503 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 22:29:22.157099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:22.157004 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:29:22.158539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:22.158505 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:29:22.848595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:22.848056 5109 patch_prober.go:29] interesting pod/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 22:29:22.848595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:22.848121 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 22:29:23.151118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:23.151021 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:29:23.151771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:23.151724 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:29:23.847154 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:23.847114 5109 patch_prober.go:29] interesting pod/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 22:29:23.847483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:23.847451 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 22:29:24.151123 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:24.151020 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:29:24.151750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:24.151713 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:29:24.509270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:24.509241 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:29:24.847221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:24.847132 5109 patch_prober.go:29] interesting pod/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 22:29:24.847480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:24.847454 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 22:29:25.900955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:25.900910 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:29:25.901942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:25.901912 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:29:25.903048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:25.903017 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:29:26.165966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:26.165858 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:26.166157 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:26.165963 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:26.416909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:26.416829 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:26.417429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:26.417403 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:26.418847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:26.418828 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" Nov 02 22:29:26.494072 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:29:26.494034 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:29:31.123272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.123234 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks"] Nov 02 22:29:31.129541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.129518 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.184148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.184114 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks"] Nov 02 22:29:31.226905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.226869 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:29:31.227116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.226929 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:31.267704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.267662 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6fb\" (UniqueName: \"kubernetes.io/projected/26cda18d-97d1-4ac5-a1aa-bd057257f683-kube-api-access-4w6fb\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.267874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.267862 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.267945 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.267885 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.369980 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.369934 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6fb\" (UniqueName: \"kubernetes.io/projected/26cda18d-97d1-4ac5-a1aa-bd057257f683-kube-api-access-4w6fb\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.370172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.370111 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.370172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.370131 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.370919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.370895 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.372779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.372751 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.431827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.431749 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6fb\" (UniqueName: \"kubernetes.io/projected/26cda18d-97d1-4ac5-a1aa-bd057257f683-kube-api-access-4w6fb\") pod \"maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.444799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.444765 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:31.825782 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:29:31.825749 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cda18d_97d1_4ac5_a1aa_bd057257f683.slice/crio-6c0a9431989fd52955d1e0e7b0d060a8c4ac12f9cad291d3ac53eb2a23273e1f WatchSource:0}: Error finding container 6c0a9431989fd52955d1e0e7b0d060a8c4ac12f9cad291d3ac53eb2a23273e1f: Status 404 returned error can't find the container with id 6c0a9431989fd52955d1e0e7b0d060a8c4ac12f9cad291d3ac53eb2a23273e1f Nov 02 22:29:31.830381 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:31.830321 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks"] Nov 02 22:29:32.749585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:32.749534 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" event={"ID":"26cda18d-97d1-4ac5-a1aa-bd057257f683","Type":"ContainerStarted","Data":"811ca4c78285f2407f7cc7592137b03c1bc8cadb03b986ff39efb1fb5ab9f2b6"} Nov 02 22:29:32.750066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:32.749615 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" event={"ID":"26cda18d-97d1-4ac5-a1aa-bd057257f683","Type":"ContainerStarted","Data":"6c0a9431989fd52955d1e0e7b0d060a8c4ac12f9cad291d3ac53eb2a23273e1f"} Nov 02 22:29:32.872703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:32.872651 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" podStartSLOduration=1.872636175 podStartE2EDuration="1.872636175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:32.86687817 +0000 UTC m=+1409.131871389" watchObservedRunningTime="2026-11-02 22:29:32.872636175 +0000 UTC m=+1409.137629393" Nov 02 22:29:33.759472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:33.759435 5109 generic.go:356] "Generic (PLEG): container finished" podID="26cda18d-97d1-4ac5-a1aa-bd057257f683" containerID="811ca4c78285f2407f7cc7592137b03c1bc8cadb03b986ff39efb1fb5ab9f2b6" exitCode=0 Nov 02 22:29:33.759878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:33.759524 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" event={"ID":"26cda18d-97d1-4ac5-a1aa-bd057257f683","Type":"ContainerDied","Data":"811ca4c78285f2407f7cc7592137b03c1bc8cadb03b986ff39efb1fb5ab9f2b6"} Nov 02 22:29:34.941517 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:34.941488 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:35.036211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.036117 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cda18d-97d1-4ac5-a1aa-bd057257f683-kube-api-access-4w6fb\" (UniqueName: \"kubernetes.io/projected/26cda18d-97d1-4ac5-a1aa-bd057257f683-kube-api-access-4w6fb\") pod \"26cda18d-97d1-4ac5-a1aa-bd057257f683\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " Nov 02 22:29:35.036370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.036299 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-openshift-ca\") pod \"26cda18d-97d1-4ac5-a1aa-bd057257f683\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " Nov 02 22:29:35.036413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.036395 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-xks-ca\") pod \"26cda18d-97d1-4ac5-a1aa-bd057257f683\" (UID: \"26cda18d-97d1-4ac5-a1aa-bd057257f683\") " Nov 02 22:29:35.036804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.036763 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-openshift-ca" pod "26cda18d-97d1-4ac5-a1aa-bd057257f683" (UID: "26cda18d-97d1-4ac5-a1aa-bd057257f683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:35.037302 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.037229 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:35.038758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.038730 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-xks-ca" pod "26cda18d-97d1-4ac5-a1aa-bd057257f683" (UID: "26cda18d-97d1-4ac5-a1aa-bd057257f683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:35.038968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.038947 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cda18d-97d1-4ac5-a1aa-bd057257f683-kube-api-access-4w6fb" pod "26cda18d-97d1-4ac5-a1aa-bd057257f683" (UID: "26cda18d-97d1-4ac5-a1aa-bd057257f683"). InnerVolumeSpecName "kube-api-access-4w6fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:35.138922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.138889 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6fb\" (UniqueName: \"kubernetes.io/projected/26cda18d-97d1-4ac5-a1aa-bd057257f683-kube-api-access-4w6fb\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:35.138922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.138924 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cda18d-97d1-4ac5-a1aa-bd057257f683-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:35.778904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.778871 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" Nov 02 22:29:35.779120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.778902 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks" event={"ID":"26cda18d-97d1-4ac5-a1aa-bd057257f683","Type":"ContainerDied","Data":"6c0a9431989fd52955d1e0e7b0d060a8c4ac12f9cad291d3ac53eb2a23273e1f"} Nov 02 22:29:35.779120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:35.778990 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0a9431989fd52955d1e0e7b0d060a8c4ac12f9cad291d3ac53eb2a23273e1f" Nov 02 22:29:39.083930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.083892 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j"] Nov 02 22:29:39.084739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.084722 5109 cpu_manager.go:414] "containerMap: removing container" podUID="26cda18d-97d1-4ac5-a1aa-bd057257f683" containerName="revoke-keys" Nov 02 22:29:39.084798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.084740 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cda18d-97d1-4ac5-a1aa-bd057257f683" containerName="revoke-keys" Nov 02 22:29:39.084884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.084874 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cda18d-97d1-4ac5-a1aa-bd057257f683" containerName="revoke-keys" Nov 02 22:29:39.089124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.089104 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.145097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.145056 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j"] Nov 02 22:29:39.187060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.186959 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:39.187060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.187049 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:29:39.223339 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.223302 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk67m\" (UniqueName: \"kubernetes.io/projected/56d2f28e-907e-42e0-a6b4-891dfdc73444-kube-api-access-fk67m\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.223540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.223367 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.223540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.223501 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.325010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.324965 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.325242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.325026 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.325311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.325245 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk67m\" (UniqueName: \"kubernetes.io/projected/56d2f28e-907e-42e0-a6b4-891dfdc73444-kube-api-access-fk67m\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.325812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.325788 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.327668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.327647 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.387239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.387151 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk67m\" (UniqueName: \"kubernetes.io/projected/56d2f28e-907e-42e0-a6b4-891dfdc73444-kube-api-access-fk67m\") pod \"maas-api-revoke-keys-e2e-ait-db36b079-88m8j\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.405329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.405280 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:39.803688 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:29:39.803646 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d2f28e_907e_42e0_a6b4_891dfdc73444.slice/crio-03f14b4c40ba9bdfe115f50d79f854eb9457d96a33d108dc1bd133cb85411cca WatchSource:0}: Error finding container 03f14b4c40ba9bdfe115f50d79f854eb9457d96a33d108dc1bd133cb85411cca: Status 404 returned error can't find the container with id 03f14b4c40ba9bdfe115f50d79f854eb9457d96a33d108dc1bd133cb85411cca Nov 02 22:29:39.809145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.809118 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j"] Nov 02 22:29:39.817738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:39.817707 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" event={"ID":"56d2f28e-907e-42e0-a6b4-891dfdc73444","Type":"ContainerStarted","Data":"03f14b4c40ba9bdfe115f50d79f854eb9457d96a33d108dc1bd133cb85411cca"} Nov 02 22:29:40.829080 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:40.829049 5109 generic.go:356] "Generic (PLEG): container finished" podID="56d2f28e-907e-42e0-a6b4-891dfdc73444" containerID="4822713d4b3d32c1874e328d8f7843cc1a9c7ba61b6ed808c379f38a9311bde8" exitCode=0 Nov 02 22:29:40.829446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:40.829128 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" event={"ID":"56d2f28e-907e-42e0-a6b4-891dfdc73444","Type":"ContainerDied","Data":"4822713d4b3d32c1874e328d8f7843cc1a9c7ba61b6ed808c379f38a9311bde8"} Nov 02 22:29:42.020806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.020776 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:42.073392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.073355 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-xks-ca\") pod \"56d2f28e-907e-42e0-a6b4-891dfdc73444\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " Nov 02 22:29:42.073392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.073396 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d2f28e-907e-42e0-a6b4-891dfdc73444-kube-api-access-fk67m\" (UniqueName: \"kubernetes.io/projected/56d2f28e-907e-42e0-a6b4-891dfdc73444-kube-api-access-fk67m\") pod \"56d2f28e-907e-42e0-a6b4-891dfdc73444\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " Nov 02 22:29:42.073697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.073684 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-openshift-ca\") pod \"56d2f28e-907e-42e0-a6b4-891dfdc73444\" (UID: \"56d2f28e-907e-42e0-a6b4-891dfdc73444\") " Nov 02 22:29:42.074056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.074029 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-openshift-ca" pod "56d2f28e-907e-42e0-a6b4-891dfdc73444" (UID: "56d2f28e-907e-42e0-a6b4-891dfdc73444"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:42.074401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.074380 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:42.075755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.075733 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-xks-ca" pod "56d2f28e-907e-42e0-a6b4-891dfdc73444" (UID: "56d2f28e-907e-42e0-a6b4-891dfdc73444"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:42.075853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.075828 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d2f28e-907e-42e0-a6b4-891dfdc73444-kube-api-access-fk67m" pod "56d2f28e-907e-42e0-a6b4-891dfdc73444" (UID: "56d2f28e-907e-42e0-a6b4-891dfdc73444"). InnerVolumeSpecName "kube-api-access-fk67m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:42.175559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.175464 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d2f28e-907e-42e0-a6b4-891dfdc73444-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:42.175559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.175495 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk67m\" (UniqueName: \"kubernetes.io/projected/56d2f28e-907e-42e0-a6b4-891dfdc73444-kube-api-access-fk67m\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:42.712368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.711530 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8"] Nov 02 22:29:42.787414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.787370 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l"] Nov 02 22:29:42.848853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.848821 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" Nov 02 22:29:42.849014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.848847 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j" event={"ID":"56d2f28e-907e-42e0-a6b4-891dfdc73444","Type":"ContainerDied","Data":"03f14b4c40ba9bdfe115f50d79f854eb9457d96a33d108dc1bd133cb85411cca"} Nov 02 22:29:42.849014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:42.848902 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f14b4c40ba9bdfe115f50d79f854eb9457d96a33d108dc1bd133cb85411cca" Nov 02 22:29:47.712000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:47.711952 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" podUID="f545a4de-d1b6-4304-8fa4-efb9dab2bc86" containerName="payload-processing" containerID="cri-o://a555e11a2e32cc0de316141fc0471832cccfad109335e4d7e8eec2d81a9e13e7" gracePeriod=25 Nov 02 22:29:47.788746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:47.788678 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" podUID="a98cd2ba-cd76-45b7-ab54-2725562ed2bd" containerName="payload-pre-processing" containerID="cri-o://3a4f5510adac619e24e1fb38cf129b8a8eb160091c1a66ffbb3d790c70df61bf" gracePeriod=25 Nov 02 22:29:47.976099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:47.975949 5109 generic.go:356] "Generic (PLEG): container finished" podID="a98cd2ba-cd76-45b7-ab54-2725562ed2bd" containerID="3a4f5510adac619e24e1fb38cf129b8a8eb160091c1a66ffbb3d790c70df61bf" exitCode=0 Nov 02 22:29:47.976502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:47.976006 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" event={"ID":"a98cd2ba-cd76-45b7-ab54-2725562ed2bd","Type":"ContainerDied","Data":"3a4f5510adac619e24e1fb38cf129b8a8eb160091c1a66ffbb3d790c70df61bf"} Nov 02 22:29:47.979540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:47.979234 5109 generic.go:356] "Generic (PLEG): container finished" podID="f545a4de-d1b6-4304-8fa4-efb9dab2bc86" containerID="a555e11a2e32cc0de316141fc0471832cccfad109335e4d7e8eec2d81a9e13e7" exitCode=0 Nov 02 22:29:47.979770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:47.979743 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" event={"ID":"f545a4de-d1b6-4304-8fa4-efb9dab2bc86","Type":"ContainerDied","Data":"a555e11a2e32cc0de316141fc0471832cccfad109335e4d7e8eec2d81a9e13e7"} Nov 02 22:29:48.208794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.208760 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:29:48.287361 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.287335 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:29:48.340143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.340107 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-plugins-config-volume\") pod \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " Nov 02 22:29:48.340554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.340528 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-plugins-config-volume" pod "a98cd2ba-cd76-45b7-ab54-2725562ed2bd" (UID: "a98cd2ba-cd76-45b7-ab54-2725562ed2bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:48.340718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.340608 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-kube-api-access-6chgw\" (UniqueName: \"kubernetes.io/projected/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-kube-api-access-6chgw\") pod \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\" (UID: \"a98cd2ba-cd76-45b7-ab54-2725562ed2bd\") " Nov 02 22:29:48.341251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.341227 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:48.343617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.343553 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-kube-api-access-6chgw" pod "a98cd2ba-cd76-45b7-ab54-2725562ed2bd" (UID: "a98cd2ba-cd76-45b7-ab54-2725562ed2bd"). InnerVolumeSpecName "kube-api-access-6chgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:48.443124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.443083 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-plugins-config-volume\") pod \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " Nov 02 22:29:48.443324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.443267 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-kube-api-access-kkllh\" (UniqueName: \"kubernetes.io/projected/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-kube-api-access-kkllh\") pod \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\" (UID: \"f545a4de-d1b6-4304-8fa4-efb9dab2bc86\") " Nov 02 22:29:48.443902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.443832 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-plugins-config-volume" pod "f545a4de-d1b6-4304-8fa4-efb9dab2bc86" (UID: "f545a4de-d1b6-4304-8fa4-efb9dab2bc86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:48.444080 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.444060 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6chgw\" (UniqueName: \"kubernetes.io/projected/a98cd2ba-cd76-45b7-ab54-2725562ed2bd-kube-api-access-6chgw\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:48.444180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.444087 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:48.446179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.446150 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-kube-api-access-kkllh" pod "f545a4de-d1b6-4304-8fa4-efb9dab2bc86" (UID: "f545a4de-d1b6-4304-8fa4-efb9dab2bc86"). InnerVolumeSpecName "kube-api-access-kkllh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:48.545295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.545208 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkllh\" (UniqueName: \"kubernetes.io/projected/f545a4de-d1b6-4304-8fa4-efb9dab2bc86-kube-api-access-kkllh\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:48.990429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.990322 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" Nov 02 22:29:48.990429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.990344 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l" event={"ID":"a98cd2ba-cd76-45b7-ab54-2725562ed2bd","Type":"ContainerDied","Data":"3e75e1d7ff24c36092956a73cd7dc404cc5d12339f0424e87a206c6fc6014b92"} Nov 02 22:29:48.991028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.990408 5109 scope.go:122] "RemoveContainer" containerID="3a4f5510adac619e24e1fb38cf129b8a8eb160091c1a66ffbb3d790c70df61bf" Nov 02 22:29:48.992539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.992519 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" Nov 02 22:29:48.992702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:48.992560 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8" event={"ID":"f545a4de-d1b6-4304-8fa4-efb9dab2bc86","Type":"ContainerDied","Data":"731118d297f7dbf3e01c55200f9d9eed077475d7befaadd813e24aaa0c83c7a6"} Nov 02 22:29:49.005009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:49.004984 5109 scope.go:122] "RemoveContainer" containerID="a555e11a2e32cc0de316141fc0471832cccfad109335e4d7e8eec2d81a9e13e7" Nov 02 22:29:49.160594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:49.160547 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8"] Nov 02 22:29:49.216500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:49.216456 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7f1542e5-5f5d98d597-7pgt8"] Nov 02 22:29:49.276640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:49.276592 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l"] Nov 02 22:29:49.282423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:49.282394 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7f1542e5-9bb48955f-5nz2l"] Nov 02 22:29:49.894075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:49.894042 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98cd2ba-cd76-45b7-ab54-2725562ed2bd" path="/var/lib/kubelet/pods/a98cd2ba-cd76-45b7-ab54-2725562ed2bd/volumes" Nov 02 22:29:49.894541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:49.894516 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f545a4de-d1b6-4304-8fa4-efb9dab2bc86" path="/var/lib/kubelet/pods/f545a4de-d1b6-4304-8fa4-efb9dab2bc86/volumes" Nov 02 22:29:56.495724 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:29:56.495685 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:29:59.062770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.062733 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g"] Nov 02 22:29:59.069045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.069015 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l"] Nov 02 22:29:59.069856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.069833 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f545a4de-d1b6-4304-8fa4-efb9dab2bc86" containerName="payload-processing" Nov 02 22:29:59.069856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.069855 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f545a4de-d1b6-4304-8fa4-efb9dab2bc86" containerName="payload-processing" Nov 02 22:29:59.070049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.069921 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a98cd2ba-cd76-45b7-ab54-2725562ed2bd" containerName="payload-pre-processing" Nov 02 22:29:59.070049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.069931 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98cd2ba-cd76-45b7-ab54-2725562ed2bd" containerName="payload-pre-processing" Nov 02 22:29:59.070049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.069946 5109 cpu_manager.go:414] "containerMap: removing container" podUID="56d2f28e-907e-42e0-a6b4-891dfdc73444" containerName="revoke-keys" Nov 02 22:29:59.070049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.069953 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d2f28e-907e-42e0-a6b4-891dfdc73444" containerName="revoke-keys" Nov 02 22:29:59.070274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.070134 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f545a4de-d1b6-4304-8fa4-efb9dab2bc86" containerName="payload-processing" Nov 02 22:29:59.070274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.070158 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98cd2ba-cd76-45b7-ab54-2725562ed2bd" containerName="payload-pre-processing" Nov 02 22:29:59.070274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.070170 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d2f28e-907e-42e0-a6b4-891dfdc73444" containerName="revoke-keys" Nov 02 22:29:59.074601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.074554 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.112494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.112458 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbec0d9b-p9r9g"] Nov 02 22:29:59.124483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.124445 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l"] Nov 02 22:29:59.127408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.127380 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:59.155975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.155943 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:29:59.196065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.196027 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn296\" (UniqueName: \"kubernetes.io/projected/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-kube-api-access-mn296\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.196279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.196182 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.196279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.196269 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.297845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.297809 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn296\" (UniqueName: \"kubernetes.io/projected/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-kube-api-access-mn296\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.298050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.297897 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.298050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.297970 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.298585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.298535 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.300546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.300523 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.361767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.361694 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn296\" (UniqueName: \"kubernetes.io/projected/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-kube-api-access-mn296\") pod \"maas-api-revoke-sub-f0e10957-c4r4l\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.390979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.390946 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:29:59.997909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.995551 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf0d71e-3de9-4424-84df-8dc916ee2196" path="/var/lib/kubelet/pods/cdf0d71e-3de9-4424-84df-8dc916ee2196/volumes" Nov 02 22:29:59.997909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.996253 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm"] Nov 02 22:29:59.997909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:29:59.996294 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-711bd4-flsqm"] Nov 02 22:30:00.165629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.165597 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h"] Nov 02 22:30:00.171757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.171731 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj"] Nov 02 22:30:00.171919 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.171870 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.177000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.176977 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.177777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.177758 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x"] Nov 02 22:30:00.183118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.183095 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.184194 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:30:00.184167 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2fce43_c9e9_45dc_9bc5_df008bbb4741.slice/crio-d0f4171bef3b5912b4c4dc42e14371b5bb61dc69ecb76463102944a39e0ad418 WatchSource:0}: Error finding container d0f4171bef3b5912b4c4dc42e14371b5bb61dc69ecb76463102944a39e0ad418: Status 404 returned error can't find the container with id d0f4171bef3b5912b4c4dc42e14371b5bb61dc69ecb76463102944a39e0ad418 Nov 02 22:30:00.185242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.185217 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9"] Nov 02 22:30:00.197484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.197456 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.219277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.219243 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l"] Nov 02 22:30:00.226655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.226628 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j"] Nov 02 22:30:00.231911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.231886 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.233171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.233152 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w"] Nov 02 22:30:00.260635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.246523 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.260635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.248419 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h"] Nov 02 22:30:00.276801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.276767 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x"] Nov 02 22:30:00.287063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.287033 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9"] Nov 02 22:30:00.297072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.297035 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj"] Nov 02 22:30:00.301741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.301712 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301bca8a-2d01-4259-a40d-5ec276afa08e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.301870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.301766 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06158047-c5ef-4601-a2eb-1f81c543870c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.301951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.301927 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9tm\" (UniqueName: \"kubernetes.io/projected/08742801-a801-461e-b6da-c0c148426c77-kube-api-access-sp9tm\") pod \"maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.302005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.301979 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5s2\" (UniqueName: \"kubernetes.io/projected/06158047-c5ef-4601-a2eb-1f81c543870c-kube-api-access-gg5s2\") pod \"maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.302047 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302034 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4edae0-0933-4c33-a0c9-fdf942973660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.302101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302087 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nwt\" (UniqueName: \"kubernetes.io/projected/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-kube-api-access-92nwt\") pod \"maas-api-key-cleanup-29894310-pzh8j\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.302136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302123 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthkx\" (UniqueName: \"kubernetes.io/projected/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-kube-api-access-mthkx\") pod \"maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.302252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302239 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-pzh8j\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.302378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302366 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgh2t\" (UniqueName: \"kubernetes.io/projected/ed4edae0-0933-4c33-a0c9-fdf942973660-kube-api-access-wgh2t\") pod \"maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.302419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302393 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.302419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302414 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm68w\" (UniqueName: \"kubernetes.io/projected/301bca8a-2d01-4259-a40d-5ec276afa08e-kube-api-access-zm68w\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.302508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.302431 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08742801-a801-461e-b6da-c0c148426c77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.306242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.306220 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j"] Nov 02 22:30:00.351341 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.351311 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w"] Nov 02 22:30:00.398423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.398384 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" event={"ID":"8b2fce43-c9e9-45dc-9bc5-df008bbb4741","Type":"ContainerStarted","Data":"d0f4171bef3b5912b4c4dc42e14371b5bb61dc69ecb76463102944a39e0ad418"} Nov 02 22:30:00.403962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.403932 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9tm\" (UniqueName: \"kubernetes.io/projected/08742801-a801-461e-b6da-c0c148426c77-kube-api-access-sp9tm\") pod \"maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.404174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.403981 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5s2\" (UniqueName: \"kubernetes.io/projected/06158047-c5ef-4601-a2eb-1f81c543870c-kube-api-access-gg5s2\") pod \"maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.404174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404122 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4edae0-0933-4c33-a0c9-fdf942973660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.404318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404189 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nwt\" (UniqueName: \"kubernetes.io/projected/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-kube-api-access-92nwt\") pod \"maas-api-key-cleanup-29894310-pzh8j\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.404318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404233 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthkx\" (UniqueName: \"kubernetes.io/projected/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-kube-api-access-mthkx\") pod \"maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.404511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404487 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-pzh8j\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.404678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404659 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgh2t\" (UniqueName: \"kubernetes.io/projected/ed4edae0-0933-4c33-a0c9-fdf942973660-kube-api-access-wgh2t\") pod \"maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.404750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404696 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.404750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404719 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm68w\" (UniqueName: \"kubernetes.io/projected/301bca8a-2d01-4259-a40d-5ec276afa08e-kube-api-access-zm68w\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.404750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404739 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08742801-a801-461e-b6da-c0c148426c77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.404920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404853 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301bca8a-2d01-4259-a40d-5ec276afa08e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.404920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.404913 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06158047-c5ef-4601-a2eb-1f81c543870c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.405070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.405046 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4edae0-0933-4c33-a0c9-fdf942973660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.405346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.405325 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-pzh8j\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.405625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.405600 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301bca8a-2d01-4259-a40d-5ec276afa08e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.405723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.405704 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.405723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.405714 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08742801-a801-461e-b6da-c0c148426c77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.405842 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.405721 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06158047-c5ef-4601-a2eb-1f81c543870c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.466088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.466048 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5s2\" (UniqueName: \"kubernetes.io/projected/06158047-c5ef-4601-a2eb-1f81c543870c-kube-api-access-gg5s2\") pod \"maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.466874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.466839 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgh2t\" (UniqueName: \"kubernetes.io/projected/ed4edae0-0933-4c33-a0c9-fdf942973660-kube-api-access-wgh2t\") pod \"maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.467067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.466877 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm68w\" (UniqueName: \"kubernetes.io/projected/301bca8a-2d01-4259-a40d-5ec276afa08e-kube-api-access-zm68w\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.467067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.466877 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthkx\" (UniqueName: \"kubernetes.io/projected/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-kube-api-access-mthkx\") pod \"maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.467197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.467120 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nwt\" (UniqueName: \"kubernetes.io/projected/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-kube-api-access-92nwt\") pod \"maas-api-key-cleanup-29894310-pzh8j\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.467760 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.467741 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9tm\" (UniqueName: \"kubernetes.io/projected/08742801-a801-461e-b6da-c0c148426c77-kube-api-access-sp9tm\") pod \"maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.510526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.510490 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:00.588644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.588611 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:00.603428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.601094 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:00.632058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.632027 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:00.638223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.638048 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:00.647446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.647420 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:00.903008 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:30:00.902976 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4edae0_0933_4c33_a0c9_fdf942973660.slice/crio-f961e7c5c8beb6c2d8a25ec0dad7fcaf14d74179acb4b568af866037309c1395 WatchSource:0}: Error finding container f961e7c5c8beb6c2d8a25ec0dad7fcaf14d74179acb4b568af866037309c1395: Status 404 returned error can't find the container with id f961e7c5c8beb6c2d8a25ec0dad7fcaf14d74179acb4b568af866037309c1395 Nov 02 22:30:00.907890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:00.907862 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h"] Nov 02 22:30:01.087760 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.087728 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj"] Nov 02 22:30:01.136561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.135834 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x"] Nov 02 22:30:01.136561 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:30:01.135856 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08742801_a801_461e_b6da_c0c148426c77.slice/crio-db4ecf38ab740a510866dfe292339d95dbbe8010f600eab9bc0067a76c992160 WatchSource:0}: Error finding container db4ecf38ab740a510866dfe292339d95dbbe8010f600eab9bc0067a76c992160: Status 404 returned error can't find the container with id db4ecf38ab740a510866dfe292339d95dbbe8010f600eab9bc0067a76c992160 Nov 02 22:30:01.144693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.144593 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j"] Nov 02 22:30:01.153776 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.153743 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9"] Nov 02 22:30:01.167780 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:30:01.167686 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e1dcad_21e3_47b0_a91f_ca1a53f037f0.slice/crio-22d48f8e5c0ccb46ae74877b322922dc40754a5e405d14e3b4e45aaf68e1403b WatchSource:0}: Error finding container 22d48f8e5c0ccb46ae74877b322922dc40754a5e405d14e3b4e45aaf68e1403b: Status 404 returned error can't find the container with id 22d48f8e5c0ccb46ae74877b322922dc40754a5e405d14e3b4e45aaf68e1403b Nov 02 22:30:01.168845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.168816 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w"] Nov 02 22:30:01.410744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.410699 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" event={"ID":"ed4edae0-0933-4c33-a0c9-fdf942973660","Type":"ContainerStarted","Data":"f961e7c5c8beb6c2d8a25ec0dad7fcaf14d74179acb4b568af866037309c1395"} Nov 02 22:30:01.412638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.412607 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" event={"ID":"8b2fce43-c9e9-45dc-9bc5-df008bbb4741","Type":"ContainerStarted","Data":"b46fdf031a70f4ffcd5a320f68cdb85686e6865687df54e468893880f882d839"} Nov 02 22:30:01.414142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.414116 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" event={"ID":"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0","Type":"ContainerStarted","Data":"22d48f8e5c0ccb46ae74877b322922dc40754a5e405d14e3b4e45aaf68e1403b"} Nov 02 22:30:01.415480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.415453 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerStarted","Data":"d557e5071e5465dd0c563a3dc5f62c403d0d4b70c4abef46654296c7312ceb8c"} Nov 02 22:30:01.416889 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.416863 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" event={"ID":"301bca8a-2d01-4259-a40d-5ec276afa08e","Type":"ContainerStarted","Data":"afcd1c99b47028222cc482c2ca8432bd12aa44ddc34686b228848b28cddadc9d"} Nov 02 22:30:01.418271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.418248 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerStarted","Data":"db4ecf38ab740a510866dfe292339d95dbbe8010f600eab9bc0067a76c992160"} Nov 02 22:30:01.419741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.419715 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" event={"ID":"94d2531f-0293-4eaa-b52d-41e23d5cf3b7","Type":"ContainerStarted","Data":"ba3caf7c7ac57690955e3bbb7d91634edcd4cd4e68c0a1b6e42c8d9e18ca201f"} Nov 02 22:30:01.530881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.530830 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" podStartSLOduration=2.530815803 podStartE2EDuration="2.530815803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:01.528908165 +0000 UTC m=+1437.793901386" watchObservedRunningTime="2026-11-02 22:30:01.530815803 +0000 UTC m=+1437.795809050" Nov 02 22:30:01.899352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:01.899261 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1775162a-b91b-4fc5-a9e4-332d9f86f9fa" path="/var/lib/kubelet/pods/1775162a-b91b-4fc5-a9e4-332d9f86f9fa/volumes" Nov 02 22:30:02.436971 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.436925 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerStarted","Data":"4dd2ffb9a4e05da785c34e2b0f6b77c17d9174d5d32d69826a2a8df0b27f6ef6"} Nov 02 22:30:02.438819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.438790 5109 generic.go:356] "Generic (PLEG): container finished" podID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerID="649d9a8869c80c14719502e6ce51908926795a30db120207abd3327f388f74d3" exitCode=60 Nov 02 22:30:02.438947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.438870 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" event={"ID":"301bca8a-2d01-4259-a40d-5ec276afa08e","Type":"ContainerDied","Data":"649d9a8869c80c14719502e6ce51908926795a30db120207abd3327f388f74d3"} Nov 02 22:30:02.439181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.439165 5109 scope.go:122] "RemoveContainer" containerID="649d9a8869c80c14719502e6ce51908926795a30db120207abd3327f388f74d3" Nov 02 22:30:02.441192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.441118 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerStarted","Data":"353ae4f2442ba2bc1166b4daecce3d33973577e5d1c4b0cbb627e2e5c408b2b6"} Nov 02 22:30:02.441368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.441352 5109 scope.go:122] "RemoveContainer" containerID="353ae4f2442ba2bc1166b4daecce3d33973577e5d1c4b0cbb627e2e5c408b2b6" Nov 02 22:30:02.443238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.443207 5109 generic.go:356] "Generic (PLEG): container finished" podID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerID="0c10ef4b1678d12fab58344af2f31cf2d6e96addcc2264a96066d686f1beb026" exitCode=60 Nov 02 22:30:02.443419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.443360 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" event={"ID":"94d2531f-0293-4eaa-b52d-41e23d5cf3b7","Type":"ContainerDied","Data":"0c10ef4b1678d12fab58344af2f31cf2d6e96addcc2264a96066d686f1beb026"} Nov 02 22:30:02.443633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.443615 5109 scope.go:122] "RemoveContainer" containerID="0c10ef4b1678d12fab58344af2f31cf2d6e96addcc2264a96066d686f1beb026" Nov 02 22:30:02.445360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.445334 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerID="1c42d491d5b67bb06af083309ceee31a5a1475f95c13f6685093f1bbd86792c6" exitCode=60 Nov 02 22:30:02.445449 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.445407 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" event={"ID":"ed4edae0-0933-4c33-a0c9-fdf942973660","Type":"ContainerDied","Data":"1c42d491d5b67bb06af083309ceee31a5a1475f95c13f6685093f1bbd86792c6"} Nov 02 22:30:02.445676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.445660 5109 scope.go:122] "RemoveContainer" containerID="1c42d491d5b67bb06af083309ceee31a5a1475f95c13f6685093f1bbd86792c6" Nov 02 22:30:02.447528 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.447505 5109 generic.go:356] "Generic (PLEG): container finished" podID="8b2fce43-c9e9-45dc-9bc5-df008bbb4741" containerID="b46fdf031a70f4ffcd5a320f68cdb85686e6865687df54e468893880f882d839" exitCode=0 Nov 02 22:30:02.447630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.447594 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" event={"ID":"8b2fce43-c9e9-45dc-9bc5-df008bbb4741","Type":"ContainerDied","Data":"b46fdf031a70f4ffcd5a320f68cdb85686e6865687df54e468893880f882d839"} Nov 02 22:30:02.449622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.449455 5109 generic.go:356] "Generic (PLEG): container finished" podID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerID="335c725477548acfdecdad1a290b0d2ab6ded9e08a9f1e7f8de83d9c7cb201b5" exitCode=60 Nov 02 22:30:02.449622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.449487 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" event={"ID":"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0","Type":"ContainerDied","Data":"335c725477548acfdecdad1a290b0d2ab6ded9e08a9f1e7f8de83d9c7cb201b5"} Nov 02 22:30:02.449880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.449863 5109 scope.go:122] "RemoveContainer" containerID="335c725477548acfdecdad1a290b0d2ab6ded9e08a9f1e7f8de83d9c7cb201b5" Nov 02 22:30:02.557390 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:02.557331 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" podStartSLOduration=2.557311992 podStartE2EDuration="2.557311992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:02.553390258 +0000 UTC m=+1438.818383482" watchObservedRunningTime="2026-11-02 22:30:02.557311992 +0000 UTC m=+1438.822305212" Nov 02 22:30:03.460965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.460859 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" event={"ID":"301bca8a-2d01-4259-a40d-5ec276afa08e","Type":"ContainerStarted","Data":"afefb7662ff136fdbf07f0566164498e860221178829fd190839b29a47dc1091"} Nov 02 22:30:03.463132 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.463097 5109 generic.go:356] "Generic (PLEG): container finished" podID="08742801-a801-461e-b6da-c0c148426c77" containerID="353ae4f2442ba2bc1166b4daecce3d33973577e5d1c4b0cbb627e2e5c408b2b6" exitCode=60 Nov 02 22:30:03.463251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.463177 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerDied","Data":"353ae4f2442ba2bc1166b4daecce3d33973577e5d1c4b0cbb627e2e5c408b2b6"} Nov 02 22:30:03.463251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.463223 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerStarted","Data":"19acf0e2bb33cefbfbeda679007762394766dba5b552b19d6c1b0938d866ae6d"} Nov 02 22:30:03.465192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.465168 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" event={"ID":"94d2531f-0293-4eaa-b52d-41e23d5cf3b7","Type":"ContainerStarted","Data":"3821d0fe1198040125bd347a1aa0cad0ecd6e0414d325d925b3fad40adba058f"} Nov 02 22:30:03.467263 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.467233 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" event={"ID":"ed4edae0-0933-4c33-a0c9-fdf942973660","Type":"ContainerStarted","Data":"8552d0ef3ff62687e5cf470b07f46689cdca825c0842ec72e31ceed5dbbaeb8b"} Nov 02 22:30:03.469271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.469240 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" event={"ID":"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0","Type":"ContainerStarted","Data":"bf885c6cb558aeab6e8d7904c9c3fc55a393772b66b357dc83cef35bab8c9b7c"} Nov 02 22:30:03.471607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.471544 5109 generic.go:356] "Generic (PLEG): container finished" podID="06158047-c5ef-4601-a2eb-1f81c543870c" containerID="4dd2ffb9a4e05da785c34e2b0f6b77c17d9174d5d32d69826a2a8df0b27f6ef6" exitCode=60 Nov 02 22:30:03.471731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.471618 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerDied","Data":"4dd2ffb9a4e05da785c34e2b0f6b77c17d9174d5d32d69826a2a8df0b27f6ef6"} Nov 02 22:30:03.471963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.471941 5109 scope.go:122] "RemoveContainer" containerID="4dd2ffb9a4e05da785c34e2b0f6b77c17d9174d5d32d69826a2a8df0b27f6ef6" Nov 02 22:30:03.583743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.583694 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" podStartSLOduration=3.5836792969999998 podStartE2EDuration="3.583679297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:03.577386047 +0000 UTC m=+1439.842379267" watchObservedRunningTime="2026-11-02 22:30:03.583679297 +0000 UTC m=+1439.848672515" Nov 02 22:30:03.695513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:03.695455 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" podStartSLOduration=3.695437561 podStartE2EDuration="3.695437561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:03.694401295 +0000 UTC m=+1439.959394526" watchObservedRunningTime="2026-11-02 22:30:03.695437561 +0000 UTC m=+1439.960430814" Nov 02 22:30:04.085101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.084996 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:30:04.244656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.232057 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" podStartSLOduration=4.232036753 podStartE2EDuration="4.232036753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:04.027365005 +0000 UTC m=+1440.292358224" watchObservedRunningTime="2026-11-02 22:30:04.232036753 +0000 UTC m=+1440.497029976" Nov 02 22:30:04.244656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.234706 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" podStartSLOduration=4.234684896 podStartE2EDuration="4.234684896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:04.150154839 +0000 UTC m=+1440.415148065" watchObservedRunningTime="2026-11-02 22:30:04.234684896 +0000 UTC m=+1440.499678116" Nov 02 22:30:04.244656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.241826 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-xks-ca\") pod \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " Nov 02 22:30:04.250029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.247903 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-kube-api-access-mn296\" (UniqueName: \"kubernetes.io/projected/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-kube-api-access-mn296\") pod \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " Nov 02 22:30:04.250029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.248012 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-openshift-ca\") pod \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\" (UID: \"8b2fce43-c9e9-45dc-9bc5-df008bbb4741\") " Nov 02 22:30:04.250029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.249631 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-xks-ca" pod "8b2fce43-c9e9-45dc-9bc5-df008bbb4741" (UID: "8b2fce43-c9e9-45dc-9bc5-df008bbb4741"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:04.264832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.264753 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-openshift-ca" pod "8b2fce43-c9e9-45dc-9bc5-df008bbb4741" (UID: "8b2fce43-c9e9-45dc-9bc5-df008bbb4741"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:04.277709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.277663 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-kube-api-access-mn296" pod "8b2fce43-c9e9-45dc-9bc5-df008bbb4741" (UID: "8b2fce43-c9e9-45dc-9bc5-df008bbb4741"). InnerVolumeSpecName "kube-api-access-mn296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:04.296106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.292740 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" podStartSLOduration=4.292718548 podStartE2EDuration="4.292718548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:04.27700917 +0000 UTC m=+1440.542002390" watchObservedRunningTime="2026-11-02 22:30:04.292718548 +0000 UTC m=+1440.557711769" Nov 02 22:30:04.350814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.350716 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn296\" (UniqueName: \"kubernetes.io/projected/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-kube-api-access-mn296\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:04.350814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.350749 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:04.350814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.350766 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2fce43-c9e9-45dc-9bc5-df008bbb4741-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:04.484345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.484299 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerStarted","Data":"ed66e52eefbdc6911d95a5604dfb0c9fa14d6d2282711bc24005a2ab1c203ae8"} Nov 02 22:30:04.486904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.486871 5109 generic.go:356] "Generic (PLEG): container finished" podID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerID="afefb7662ff136fdbf07f0566164498e860221178829fd190839b29a47dc1091" exitCode=60 Nov 02 22:30:04.487093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.486965 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" event={"ID":"301bca8a-2d01-4259-a40d-5ec276afa08e","Type":"ContainerDied","Data":"afefb7662ff136fdbf07f0566164498e860221178829fd190839b29a47dc1091"} Nov 02 22:30:04.487093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.487037 5109 scope.go:122] "RemoveContainer" containerID="649d9a8869c80c14719502e6ce51908926795a30db120207abd3327f388f74d3" Nov 02 22:30:04.487803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.487319 5109 scope.go:122] "RemoveContainer" containerID="afefb7662ff136fdbf07f0566164498e860221178829fd190839b29a47dc1091" Nov 02 22:30:04.487872 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:04.487851 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj_odh-ai-gateway-infra(301bca8a-2d01-4259-a40d-5ec276afa08e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.490141 5109 generic.go:356] "Generic (PLEG): container finished" podID="08742801-a801-461e-b6da-c0c148426c77" containerID="19acf0e2bb33cefbfbeda679007762394766dba5b552b19d6c1b0938d866ae6d" exitCode=60 Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.490245 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerDied","Data":"19acf0e2bb33cefbfbeda679007762394766dba5b552b19d6c1b0938d866ae6d"} Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.490536 5109 scope.go:122] "RemoveContainer" containerID="19acf0e2bb33cefbfbeda679007762394766dba5b552b19d6c1b0938d866ae6d" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:04.491004 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9_odh-ai-gateway-infra(08742801-a801-461e-b6da-c0c148426c77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" podUID="08742801-a801-461e-b6da-c0c148426c77" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.492996 5109 generic.go:356] "Generic (PLEG): container finished" podID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerID="3821d0fe1198040125bd347a1aa0cad0ecd6e0414d325d925b3fad40adba058f" exitCode=60 Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.493083 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" event={"ID":"94d2531f-0293-4eaa-b52d-41e23d5cf3b7","Type":"ContainerDied","Data":"3821d0fe1198040125bd347a1aa0cad0ecd6e0414d325d925b3fad40adba058f"} Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.493420 5109 scope.go:122] "RemoveContainer" containerID="3821d0fe1198040125bd347a1aa0cad0ecd6e0414d325d925b3fad40adba058f" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:04.493839 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-pzh8j_odh-ai-gateway-infra(94d2531f-0293-4eaa-b52d-41e23d5cf3b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.495738 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerID="8552d0ef3ff62687e5cf470b07f46689cdca825c0842ec72e31ceed5dbbaeb8b" exitCode=60 Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.495819 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" event={"ID":"ed4edae0-0933-4c33-a0c9-fdf942973660","Type":"ContainerDied","Data":"8552d0ef3ff62687e5cf470b07f46689cdca825c0842ec72e31ceed5dbbaeb8b"} Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.496101 5109 scope.go:122] "RemoveContainer" containerID="8552d0ef3ff62687e5cf470b07f46689cdca825c0842ec72e31ceed5dbbaeb8b" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:04.496483 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h_odh-ai-gateway-infra(ed4edae0-0933-4c33-a0c9-fdf942973660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.497307 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.497305 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l" event={"ID":"8b2fce43-c9e9-45dc-9bc5-df008bbb4741","Type":"ContainerDied","Data":"d0f4171bef3b5912b4c4dc42e14371b5bb61dc69ecb76463102944a39e0ad418"} Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.497423 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f4171bef3b5912b4c4dc42e14371b5bb61dc69ecb76463102944a39e0ad418" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.499354 5109 generic.go:356] "Generic (PLEG): container finished" podID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerID="bf885c6cb558aeab6e8d7904c9c3fc55a393772b66b357dc83cef35bab8c9b7c" exitCode=60 Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.499378 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" event={"ID":"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0","Type":"ContainerDied","Data":"bf885c6cb558aeab6e8d7904c9c3fc55a393772b66b357dc83cef35bab8c9b7c"} Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.499689 5109 scope.go:122] "RemoveContainer" containerID="bf885c6cb558aeab6e8d7904c9c3fc55a393772b66b357dc83cef35bab8c9b7c" Nov 02 22:30:04.502599 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:04.499982 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w_odh-ai-gateway-infra(c8e1dcad-21e3-47b0-a91f-ca1a53f037f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" Nov 02 22:30:04.520792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.520770 5109 scope.go:122] "RemoveContainer" containerID="353ae4f2442ba2bc1166b4daecce3d33973577e5d1c4b0cbb627e2e5c408b2b6" Nov 02 22:30:04.535743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.535720 5109 scope.go:122] "RemoveContainer" containerID="0c10ef4b1678d12fab58344af2f31cf2d6e96addcc2264a96066d686f1beb026" Nov 02 22:30:04.550766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.550747 5109 scope.go:122] "RemoveContainer" containerID="1c42d491d5b67bb06af083309ceee31a5a1475f95c13f6685093f1bbd86792c6" Nov 02 22:30:04.565804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:04.565782 5109 scope.go:122] "RemoveContainer" containerID="335c725477548acfdecdad1a290b0d2ab6ded9e08a9f1e7f8de83d9c7cb201b5" Nov 02 22:30:05.511516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.511485 5109 scope.go:122] "RemoveContainer" containerID="bf885c6cb558aeab6e8d7904c9c3fc55a393772b66b357dc83cef35bab8c9b7c" Nov 02 22:30:05.512008 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:05.511934 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w_odh-ai-gateway-infra(c8e1dcad-21e3-47b0-a91f-ca1a53f037f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" Nov 02 22:30:05.513559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.513533 5109 generic.go:356] "Generic (PLEG): container finished" podID="06158047-c5ef-4601-a2eb-1f81c543870c" containerID="ed66e52eefbdc6911d95a5604dfb0c9fa14d6d2282711bc24005a2ab1c203ae8" exitCode=60 Nov 02 22:30:05.513739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.513594 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerDied","Data":"ed66e52eefbdc6911d95a5604dfb0c9fa14d6d2282711bc24005a2ab1c203ae8"} Nov 02 22:30:05.513739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.513654 5109 scope.go:122] "RemoveContainer" containerID="4dd2ffb9a4e05da785c34e2b0f6b77c17d9174d5d32d69826a2a8df0b27f6ef6" Nov 02 22:30:05.514059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.514034 5109 scope.go:122] "RemoveContainer" containerID="ed66e52eefbdc6911d95a5604dfb0c9fa14d6d2282711bc24005a2ab1c203ae8" Nov 02 22:30:05.514510 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:05.514479 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x_odh-ai-gateway-infra(06158047-c5ef-4601-a2eb-1f81c543870c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" Nov 02 22:30:05.515975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.515958 5109 scope.go:122] "RemoveContainer" containerID="afefb7662ff136fdbf07f0566164498e860221178829fd190839b29a47dc1091" Nov 02 22:30:05.516340 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:05.516321 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj_odh-ai-gateway-infra(301bca8a-2d01-4259-a40d-5ec276afa08e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" Nov 02 22:30:05.517814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.517793 5109 scope.go:122] "RemoveContainer" containerID="19acf0e2bb33cefbfbeda679007762394766dba5b552b19d6c1b0938d866ae6d" Nov 02 22:30:05.518201 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:05.518176 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9_odh-ai-gateway-infra(08742801-a801-461e-b6da-c0c148426c77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" podUID="08742801-a801-461e-b6da-c0c148426c77" Nov 02 22:30:05.519678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.519658 5109 scope.go:122] "RemoveContainer" containerID="3821d0fe1198040125bd347a1aa0cad0ecd6e0414d325d925b3fad40adba058f" Nov 02 22:30:05.519998 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:05.519980 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-pzh8j_odh-ai-gateway-infra(94d2531f-0293-4eaa-b52d-41e23d5cf3b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" Nov 02 22:30:05.521599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:05.521581 5109 scope.go:122] "RemoveContainer" containerID="8552d0ef3ff62687e5cf470b07f46689cdca825c0842ec72e31ceed5dbbaeb8b" Nov 02 22:30:05.521886 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:05.521867 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h_odh-ai-gateway-infra(ed4edae0-0933-4c33-a0c9-fdf942973660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" Nov 02 22:30:06.449894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.449854 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g"] Nov 02 22:30:06.451277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.451252 5109 cpu_manager.go:414] "containerMap: removing container" podUID="8b2fce43-c9e9-45dc-9bc5-df008bbb4741" containerName="revoke-keys" Nov 02 22:30:06.451277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.451278 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2fce43-c9e9-45dc-9bc5-df008bbb4741" containerName="revoke-keys" Nov 02 22:30:06.451480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.451469 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2fce43-c9e9-45dc-9bc5-df008bbb4741" containerName="revoke-keys" Nov 02 22:30:06.456584 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.456549 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.503009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.502973 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.503202 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.503109 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxcr\" (UniqueName: \"kubernetes.io/projected/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-kube-api-access-zsxcr\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.503377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.503346 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.508404 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.508372 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g"] Nov 02 22:30:06.508939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.508915 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:30:06.534509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.534479 5109 scope.go:122] "RemoveContainer" containerID="ed66e52eefbdc6911d95a5604dfb0c9fa14d6d2282711bc24005a2ab1c203ae8" Nov 02 22:30:06.535053 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:06.534910 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x_odh-ai-gateway-infra(06158047-c5ef-4601-a2eb-1f81c543870c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" Nov 02 22:30:06.605264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.605217 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.605441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.605365 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.605441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.605410 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxcr\" (UniqueName: \"kubernetes.io/projected/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-kube-api-access-zsxcr\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.605956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.605930 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.608531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.608501 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.668223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.668190 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxcr\" (UniqueName: \"kubernetes.io/projected/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-kube-api-access-zsxcr\") pod \"maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.775603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.773944 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:06.983410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.983372 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl"] Nov 02 22:30:06.983733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.983666 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" containerID="cri-o://a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d" gracePeriod=30 Nov 02 22:30:06.991840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.991808 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm"] Nov 02 22:30:06.992024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.992000 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" containerID="cri-o://bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e" gracePeriod=30 Nov 02 22:30:06.999633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.999600 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj"] Nov 02 22:30:06.999841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:06.999815 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" containerID="cri-o://e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d" gracePeriod=30 Nov 02 22:30:07.022404 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:07.022355 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9"] Nov 02 22:30:07.022674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:07.022650 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" containerID="cri-o://2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d" gracePeriod=30 Nov 02 22:30:07.242678 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:30:07.242636 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7c288a_4177_4f3b_a975_d5cbbf7ffae3.slice/crio-5fc42d3998231b7fdfe8797149f625ac9ddb88bc1ebc16cc7e30c4fe7c6ad3dd WatchSource:0}: Error finding container 5fc42d3998231b7fdfe8797149f625ac9ddb88bc1ebc16cc7e30c4fe7c6ad3dd: Status 404 returned error can't find the container with id 5fc42d3998231b7fdfe8797149f625ac9ddb88bc1ebc16cc7e30c4fe7c6ad3dd Nov 02 22:30:07.421915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:07.421881 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g"] Nov 02 22:30:07.546170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:07.546128 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" event={"ID":"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3","Type":"ContainerStarted","Data":"5fc42d3998231b7fdfe8797149f625ac9ddb88bc1ebc16cc7e30c4fe7c6ad3dd"} Nov 02 22:30:08.559769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:08.559725 5109 generic.go:356] "Generic (PLEG): container finished" podID="9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" containerID="e6859f3b1c2a8bd406678118c60e8ec82fb5d41f4d1c8bb2de25a70ca6bc1241" exitCode=0 Nov 02 22:30:08.560249 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:08.559790 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" event={"ID":"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3","Type":"ContainerDied","Data":"e6859f3b1c2a8bd406678118c60e8ec82fb5d41f4d1c8bb2de25a70ca6bc1241"} Nov 02 22:30:09.092775 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.092734 5109 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:30:09.092965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.092800 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:30:09.764979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.764946 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:09.885986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.885889 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-kube-api-access-zsxcr\" (UniqueName: \"kubernetes.io/projected/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-kube-api-access-zsxcr\") pod \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " Nov 02 22:30:09.885986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.885946 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-openshift-ca\") pod \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " Nov 02 22:30:09.886225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.886179 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-xks-ca\") pod \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\" (UID: \"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3\") " Nov 02 22:30:09.886360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.886334 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-openshift-ca" pod "9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" (UID: "9d7c288a-4177-4f3b-a975-d5cbbf7ffae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:09.886953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.886930 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:09.888876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.888845 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-xks-ca" pod "9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" (UID: "9d7c288a-4177-4f3b-a975-d5cbbf7ffae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:09.888876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.888867 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-kube-api-access-zsxcr" pod "9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" (UID: "9d7c288a-4177-4f3b-a975-d5cbbf7ffae3"). InnerVolumeSpecName "kube-api-access-zsxcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:09.988655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.988621 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:09.988655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:09.988654 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxcr\" (UniqueName: \"kubernetes.io/projected/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3-kube-api-access-zsxcr\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:10.580535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:10.580504 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" Nov 02 22:30:10.580535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:10.580519 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g" event={"ID":"9d7c288a-4177-4f3b-a975-d5cbbf7ffae3","Type":"ContainerDied","Data":"5fc42d3998231b7fdfe8797149f625ac9ddb88bc1ebc16cc7e30c4fe7c6ad3dd"} Nov 02 22:30:10.580800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:10.580606 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fc42d3998231b7fdfe8797149f625ac9ddb88bc1ebc16cc7e30c4fe7c6ad3dd" Nov 02 22:30:12.369381 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.369348 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:30:12.410876 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.410851 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:30:12.433064 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.433035 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:30:12.438773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.438741 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:30:12.464368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464316 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f4ec82f-1f47-449f-8c45-244460554d96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4ec82f-1f47-449f-8c45-244460554d96-istio-podinfo\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.464584 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464384 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-kube-api-access-24mkp\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-kube-api-access-24mkp\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.464584 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464407 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7802e21-b667-4a85-8e00-78b67e4a2548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7802e21-b667-4a85-8e00-78b67e4a2548-istiod-ca-cert\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.464584 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464501 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-kube-api-access-qxn4f\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-kube-api-access-qxn4f\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.464584 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464539 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-certs\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.464808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464620 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-socket\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.464808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464714 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7539982-c036-4526-9d1f-bc9324bed23f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7539982-c036-4526-9d1f-bc9324bed23f-istiod-ca-cert\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.464808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464752 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-data\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464815 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-envoy\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464853 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-envoy\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464916 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-certs\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464946 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-podinfo\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464946 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-certs" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.464977 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7802e21-b667-4a85-8e00-78b67e4a2548-istiod-ca-cert" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465022 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-credential-socket\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465053 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-data\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465093 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4ec82f-1f47-449f-8c45-244460554d96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4ec82f-1f47-449f-8c45-244460554d96-istiod-ca-cert\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465149 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-kube-api-access-jnsqs\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-kube-api-access-jnsqs\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465196 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-certs\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465238 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-istio-token\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-istio-token\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.465284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465266 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-kube-api-access-7b4j6\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-kube-api-access-7b4j6\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465324 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-certs\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465360 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istiod-ca-cert\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465394 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-token\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-token\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465389 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7539982-c036-4526-9d1f-bc9324bed23f-istiod-ca-cert" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465428 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-data\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465469 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-token\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465503 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-envoy\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465558 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-istio-token\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-istio-token\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465653 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-credential-socket\") pod \"6f4ec82f-1f47-449f-8c45-244460554d96\" (UID: \"6f4ec82f-1f47-449f-8c45-244460554d96\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465662 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-socket" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465695 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-podinfo\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465773 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-credential-socket\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465872 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-socket\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465891 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-certs" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465900 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7539982-c036-4526-9d1f-bc9324bed23f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7539982-c036-4526-9d1f-bc9324bed23f-istio-podinfo\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465943 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-socket\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.465985 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-socket\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.466038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.466040 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-credential-socket" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.467182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.466060 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-credential-socket\") pod \"d7539982-c036-4526-9d1f-bc9324bed23f\" (UID: \"d7539982-c036-4526-9d1f-bc9324bed23f\") " Nov 02 22:30:12.467182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.466117 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-envoy\") pod \"e7802e21-b667-4a85-8e00-78b67e4a2548\" (UID: \"e7802e21-b667-4a85-8e00-78b67e4a2548\") " Nov 02 22:30:12.467182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.466153 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-data" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.467182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.466167 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-data\") pod \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\" (UID: \"0e17753b-68c5-4a7b-a87b-cd0275a77b1f\") " Nov 02 22:30:12.467182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.466337 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4ec82f-1f47-449f-8c45-244460554d96-istiod-ca-cert" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:12.467459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467302 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.467459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467325 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4ec82f-1f47-449f-8c45-244460554d96-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.467459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467341 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.467459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467356 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7802e21-b667-4a85-8e00-78b67e4a2548-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.467459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467370 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.467459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467385 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.467459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467397 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7539982-c036-4526-9d1f-bc9324bed23f-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.467836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467612 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f4ec82f-1f47-449f-8c45-244460554d96-istio-podinfo" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:12.467836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467659 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-data" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.467836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467689 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-kube-api-access-qxn4f" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "kube-api-access-qxn4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.467836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467806 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-data" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467831 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-envoy" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467839 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-podinfo" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:12.468040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.467956 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-certs" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468087 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-credential-socket" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468088 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-socket" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468099 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-credential-socket" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468425 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468277 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-socket" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468425 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468283 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-socket" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468425 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468300 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-credential-socket" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468466 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-data" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.468719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.468697 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-certs" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.469182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.469157 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istiod-ca-cert" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:12.470143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.469874 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-envoy" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.470143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.469972 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-kube-api-access-jnsqs" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "kube-api-access-jnsqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.470810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.470783 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7539982-c036-4526-9d1f-bc9324bed23f-istio-podinfo" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:12.470810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.470805 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-kube-api-access-7b4j6" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "kube-api-access-7b4j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.471268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.471247 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-istio-token" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.471355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.471285 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-kube-api-access-24mkp" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "kube-api-access-24mkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.472805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.471752 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-token" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.472805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.472682 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-envoy" pod "e7802e21-b667-4a85-8e00-78b67e4a2548" (UID: "e7802e21-b667-4a85-8e00-78b67e4a2548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.472805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.472761 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-podinfo" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:12.472805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.472777 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-envoy" pod "6f4ec82f-1f47-449f-8c45-244460554d96" (UID: "6f4ec82f-1f47-449f-8c45-244460554d96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.473323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.473303 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-token" pod "0e17753b-68c5-4a7b-a87b-cd0275a77b1f" (UID: "0e17753b-68c5-4a7b-a87b-cd0275a77b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.473323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.473312 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-istio-token" pod "d7539982-c036-4526-9d1f-bc9324bed23f" (UID: "d7539982-c036-4526-9d1f-bc9324bed23f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.569324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569231 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569269 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569285 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7539982-c036-4526-9d1f-bc9324bed23f-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569298 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569312 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569328 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569343 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569357 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569369 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4ec82f-1f47-449f-8c45-244460554d96-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569381 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24mkp\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-kube-api-access-24mkp\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569396 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn4f\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-kube-api-access-qxn4f\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569409 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569421 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569435 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569448 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7539982-c036-4526-9d1f-bc9324bed23f-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569461 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569475 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569488 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsqs\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-kube-api-access-jnsqs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569502 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4ec82f-1f47-449f-8c45-244460554d96-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569517 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b4j6\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-kube-api-access-7b4j6\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569532 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569547 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569561 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569596 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7802e21-b667-4a85-8e00-78b67e4a2548-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569609 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569625 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569639 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7539982-c036-4526-9d1f-bc9324bed23f-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569654 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4ec82f-1f47-449f-8c45-244460554d96-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.569730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.569669 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17753b-68c5-4a7b-a87b-cd0275a77b1f-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.602066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.602025 5109 generic.go:356] "Generic (PLEG): container finished" podID="d7539982-c036-4526-9d1f-bc9324bed23f" containerID="a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d" exitCode=0 Nov 02 22:30:12.602251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.602097 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" Nov 02 22:30:12.602251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.602119 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" event={"ID":"d7539982-c036-4526-9d1f-bc9324bed23f","Type":"ContainerDied","Data":"a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d"} Nov 02 22:30:12.602251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.602203 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl" event={"ID":"d7539982-c036-4526-9d1f-bc9324bed23f","Type":"ContainerDied","Data":"1baa195c76b5bbab0e01e769dab2378607871b7008c51942e15d2d1fb446a578"} Nov 02 22:30:12.602251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.602231 5109 scope.go:122] "RemoveContainer" containerID="a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d" Nov 02 22:30:12.604076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.604049 5109 generic.go:356] "Generic (PLEG): container finished" podID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerID="2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d" exitCode=0 Nov 02 22:30:12.604199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.604099 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" Nov 02 22:30:12.604199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.604172 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" event={"ID":"0e17753b-68c5-4a7b-a87b-cd0275a77b1f","Type":"ContainerDied","Data":"2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d"} Nov 02 22:30:12.604312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.604214 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9" event={"ID":"0e17753b-68c5-4a7b-a87b-cd0275a77b1f","Type":"ContainerDied","Data":"9716992e860f8c5810c606a09e76d4d57c3276d8f97ae3c6a7efc7e78ff0e826"} Nov 02 22:30:12.623800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.623774 5109 scope.go:122] "RemoveContainer" containerID="bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d" Nov 02 22:30:12.624536 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.624511 5109 generic.go:356] "Generic (PLEG): container finished" podID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerID="bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e" exitCode=0 Nov 02 22:30:12.624755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.624733 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" event={"ID":"e7802e21-b667-4a85-8e00-78b67e4a2548","Type":"ContainerDied","Data":"bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e"} Nov 02 22:30:12.624857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.624789 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" event={"ID":"e7802e21-b667-4a85-8e00-78b67e4a2548","Type":"ContainerDied","Data":"d98bfd90b81fd273870753710ae14b0f969b0a186465c656ba4308e0660e68da"} Nov 02 22:30:12.625028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.625008 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm" Nov 02 22:30:12.638037 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.637957 5109 generic.go:356] "Generic (PLEG): container finished" podID="6f4ec82f-1f47-449f-8c45-244460554d96" containerID="e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d" exitCode=0 Nov 02 22:30:12.638557 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.638198 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" event={"ID":"6f4ec82f-1f47-449f-8c45-244460554d96","Type":"ContainerDied","Data":"e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d"} Nov 02 22:30:12.638557 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.638270 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" event={"ID":"6f4ec82f-1f47-449f-8c45-244460554d96","Type":"ContainerDied","Data":"7eb573e0d9bdd18c785361ebc566e9d7030ce3597a47d64da6921fc848afa9a4"} Nov 02 22:30:12.638557 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.638368 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj" Nov 02 22:30:12.645825 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.645774 5109 scope.go:122] "RemoveContainer" containerID="a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d" Nov 02 22:30:12.646248 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:12.646217 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d\": container with ID starting with a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d not found: ID does not exist" containerID="a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d" Nov 02 22:30:12.646347 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.646263 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d"} err="failed to get container status \"a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d\": rpc error: code = NotFound desc = could not find container \"a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d\": container with ID starting with a721cfc61cb67058473094a560e0c40b3df9910435ba0045da97681a2da3152d not found: ID does not exist" Nov 02 22:30:12.646347 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.646287 5109 scope.go:122] "RemoveContainer" containerID="bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d" Nov 02 22:30:12.646704 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:12.646680 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d\": container with ID starting with bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d not found: ID does not exist" containerID="bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d" Nov 02 22:30:12.646808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.646711 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d"} err="failed to get container status \"bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d\": rpc error: code = NotFound desc = could not find container \"bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d\": container with ID starting with bf1901e181b07503bc8a04e619a25fde9c33cc283508c3219390af9dea1ad95d not found: ID does not exist" Nov 02 22:30:12.646808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.646725 5109 scope.go:122] "RemoveContainer" containerID="2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d" Nov 02 22:30:12.665791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.665760 5109 scope.go:122] "RemoveContainer" containerID="2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d" Nov 02 22:30:12.667123 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:12.666796 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d\": container with ID starting with 2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d not found: ID does not exist" containerID="2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d" Nov 02 22:30:12.667123 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.666853 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d"} err="failed to get container status \"2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d\": rpc error: code = NotFound desc = could not find container \"2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d\": container with ID starting with 2d47a8e02ed1f0dffcd18b6d5a109b5c328bcc647c7c240050b817df5d940d7d not found: ID does not exist" Nov 02 22:30:12.667123 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.666880 5109 scope.go:122] "RemoveContainer" containerID="bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e" Nov 02 22:30:12.684000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.683977 5109 scope.go:122] "RemoveContainer" containerID="bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e" Nov 02 22:30:12.684402 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:12.684377 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e\": container with ID starting with bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e not found: ID does not exist" containerID="bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e" Nov 02 22:30:12.684473 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.684412 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e"} err="failed to get container status \"bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e\": rpc error: code = NotFound desc = could not find container \"bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e\": container with ID starting with bac0c6ea43262c05de6d131cea8519746111f55a797412241182c166906b487e not found: ID does not exist" Nov 02 22:30:12.684473 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.684429 5109 scope.go:122] "RemoveContainer" containerID="e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d" Nov 02 22:30:12.698792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.698766 5109 scope.go:122] "RemoveContainer" containerID="e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d" Nov 02 22:30:12.699190 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:12.699158 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d\": container with ID starting with e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d not found: ID does not exist" containerID="e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d" Nov 02 22:30:12.699301 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:12.699203 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d"} err="failed to get container status \"e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d\": rpc error: code = NotFound desc = could not find container \"e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d\": container with ID starting with e2999e5028b04aff59e04e40f800caa74a81604b25fd13ed025c46c069352b9d not found: ID does not exist" Nov 02 22:30:13.024738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.024703 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj"] Nov 02 22:30:13.031433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.031401 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-s4jvj"] Nov 02 22:30:13.140722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.140691 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl"] Nov 02 22:30:13.146375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.146351 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-l6kzl"] Nov 02 22:30:13.257857 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.257815 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9"] Nov 02 22:30:13.264692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.264615 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-2vbg9"] Nov 02 22:30:13.375349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.375315 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm"] Nov 02 22:30:13.380769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.380746 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-p2rzm"] Nov 02 22:30:13.894946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.894914 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" path="/var/lib/kubelet/pods/0e17753b-68c5-4a7b-a87b-cd0275a77b1f/volumes" Nov 02 22:30:13.895405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.895388 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" path="/var/lib/kubelet/pods/6f4ec82f-1f47-449f-8c45-244460554d96/volumes" Nov 02 22:30:13.895869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.895854 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" path="/var/lib/kubelet/pods/d7539982-c036-4526-9d1f-bc9324bed23f/volumes" Nov 02 22:30:13.896653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:13.896639 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" path="/var/lib/kubelet/pods/e7802e21-b667-4a85-8e00-78b67e4a2548/volumes" Nov 02 22:30:14.395851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:14.395807 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w"] Nov 02 22:30:14.898908 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:14.898877 5109 scope.go:122] "RemoveContainer" containerID="8552d0ef3ff62687e5cf470b07f46689cdca825c0842ec72e31ceed5dbbaeb8b" Nov 02 22:30:15.002336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.001088 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:15.213524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.213487 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-kube-api-access-mthkx\" (UniqueName: \"kubernetes.io/projected/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-kube-api-access-mthkx\") pod \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " Nov 02 22:30:15.213768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.213748 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-maas-api-service-ca\") pod \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\" (UID: \"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0\") " Nov 02 22:30:15.214133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.214107 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-maas-api-service-ca" pod "c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" (UID: "c8e1dcad-21e3-47b0-a91f-ca1a53f037f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:15.214412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.214393 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:15.224498 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.224471 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-kube-api-access-mthkx" pod "c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" (UID: "c8e1dcad-21e3-47b0-a91f-ca1a53f037f0"). InnerVolumeSpecName "kube-api-access-mthkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:15.315419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.315382 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mthkx\" (UniqueName: \"kubernetes.io/projected/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0-kube-api-access-mthkx\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:15.706357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.706325 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" event={"ID":"ed4edae0-0933-4c33-a0c9-fdf942973660","Type":"ContainerStarted","Data":"cd714cbfd8291d3ba7bced7d5efd4e4f18d40d4336c5197c6f8eae74a96eb382"} Nov 02 22:30:15.709508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.709480 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" event={"ID":"c8e1dcad-21e3-47b0-a91f-ca1a53f037f0","Type":"ContainerDied","Data":"22d48f8e5c0ccb46ae74877b322922dc40754a5e405d14e3b4e45aaf68e1403b"} Nov 02 22:30:15.709755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.709655 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w" Nov 02 22:30:15.709900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.709885 5109 scope.go:122] "RemoveContainer" containerID="bf885c6cb558aeab6e8d7904c9c3fc55a393772b66b357dc83cef35bab8c9b7c" Nov 02 22:30:15.885540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.885509 5109 scope.go:122] "RemoveContainer" containerID="ed66e52eefbdc6911d95a5604dfb0c9fa14d6d2282711bc24005a2ab1c203ae8" Nov 02 22:30:15.943196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.943163 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w"] Nov 02 22:30:15.949151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:15.949121 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7f1542e5-29894310-9ds2w"] Nov 02 22:30:16.707214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.706751 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks"] Nov 02 22:30:16.715661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.715630 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f1542e5-qr4ks"] Nov 02 22:30:16.720748 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.720718 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerID="cd714cbfd8291d3ba7bced7d5efd4e4f18d40d4336c5197c6f8eae74a96eb382" exitCode=60 Nov 02 22:30:16.720881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.720790 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" event={"ID":"ed4edae0-0933-4c33-a0c9-fdf942973660","Type":"ContainerDied","Data":"cd714cbfd8291d3ba7bced7d5efd4e4f18d40d4336c5197c6f8eae74a96eb382"} Nov 02 22:30:16.720952 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.720881 5109 scope.go:122] "RemoveContainer" containerID="8552d0ef3ff62687e5cf470b07f46689cdca825c0842ec72e31ceed5dbbaeb8b" Nov 02 22:30:16.721362 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.721331 5109 scope.go:122] "RemoveContainer" containerID="cd714cbfd8291d3ba7bced7d5efd4e4f18d40d4336c5197c6f8eae74a96eb382" Nov 02 22:30:16.721848 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:16.721821 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h_odh-ai-gateway-infra(ed4edae0-0933-4c33-a0c9-fdf942973660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" Nov 02 22:30:16.727984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.727938 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerStarted","Data":"dfa8e334293ef863a4a3936f867d240fd099c078ad811eb99a83b83cab1d6509"} Nov 02 22:30:16.842050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:16.841916 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h"] Nov 02 22:30:17.846605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:17.838972 5109 generic.go:356] "Generic (PLEG): container finished" podID="06158047-c5ef-4601-a2eb-1f81c543870c" containerID="dfa8e334293ef863a4a3936f867d240fd099c078ad811eb99a83b83cab1d6509" exitCode=60 Nov 02 22:30:17.851538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:17.839186 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerDied","Data":"dfa8e334293ef863a4a3936f867d240fd099c078ad811eb99a83b83cab1d6509"} Nov 02 22:30:17.851884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:17.851861 5109 scope.go:122] "RemoveContainer" containerID="ed66e52eefbdc6911d95a5604dfb0c9fa14d6d2282711bc24005a2ab1c203ae8" Nov 02 22:30:17.852663 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:17.852638 5109 scope.go:122] "RemoveContainer" containerID="dfa8e334293ef863a4a3936f867d240fd099c078ad811eb99a83b83cab1d6509" Nov 02 22:30:17.853446 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:17.853414 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x_odh-ai-gateway-infra(06158047-c5ef-4601-a2eb-1f81c543870c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" Nov 02 22:30:17.902605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:17.890343 5109 scope.go:122] "RemoveContainer" containerID="afefb7662ff136fdbf07f0566164498e860221178829fd190839b29a47dc1091" Nov 02 22:30:17.903316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:17.903288 5109 scope.go:122] "RemoveContainer" containerID="3821d0fe1198040125bd347a1aa0cad0ecd6e0414d325d925b3fad40adba058f" Nov 02 22:30:18.100831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.060865 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cda18d-97d1-4ac5-a1aa-bd057257f683" path="/var/lib/kubelet/pods/26cda18d-97d1-4ac5-a1aa-bd057257f683/volumes" Nov 02 22:30:18.100831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.082018 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" path="/var/lib/kubelet/pods/c8e1dcad-21e3-47b0-a91f-ca1a53f037f0/volumes" Nov 02 22:30:18.100831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.082644 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x"] Nov 02 22:30:18.237103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.237073 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc"] Nov 02 22:30:18.357498 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.357435 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:18.447250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.447223 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4edae0-0933-4c33-a0c9-fdf942973660-kube-api-access-wgh2t\" (UniqueName: \"kubernetes.io/projected/ed4edae0-0933-4c33-a0c9-fdf942973660-kube-api-access-wgh2t\") pod \"ed4edae0-0933-4c33-a0c9-fdf942973660\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " Nov 02 22:30:18.447370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.447289 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4edae0-0933-4c33-a0c9-fdf942973660-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4edae0-0933-4c33-a0c9-fdf942973660-maas-api-service-ca\") pod \"ed4edae0-0933-4c33-a0c9-fdf942973660\" (UID: \"ed4edae0-0933-4c33-a0c9-fdf942973660\") " Nov 02 22:30:18.447659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.447639 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4edae0-0933-4c33-a0c9-fdf942973660-maas-api-service-ca" pod "ed4edae0-0933-4c33-a0c9-fdf942973660" (UID: "ed4edae0-0933-4c33-a0c9-fdf942973660"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:18.447771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.447754 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4edae0-0933-4c33-a0c9-fdf942973660-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:18.449537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.449518 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4edae0-0933-4c33-a0c9-fdf942973660-kube-api-access-wgh2t" pod "ed4edae0-0933-4c33-a0c9-fdf942973660" (UID: "ed4edae0-0933-4c33-a0c9-fdf942973660"). InnerVolumeSpecName "kube-api-access-wgh2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:18.465933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.465907 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8"] Nov 02 22:30:18.466696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466681 5109 cpu_manager.go:414] "containerMap: removing container" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:18.466736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466697 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:18.466736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466711 5109 cpu_manager.go:414] "containerMap: removing container" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" Nov 02 22:30:18.466736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466717 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" Nov 02 22:30:18.466736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466727 5109 cpu_manager.go:414] "containerMap: removing container" podUID="9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" containerName="revoke-keys" Nov 02 22:30:18.466736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466732 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" containerName="revoke-keys" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466740 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466745 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466760 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerName="cleanup" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466766 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerName="cleanup" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466780 5109 cpu_manager.go:414] "containerMap: removing container" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466784 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466793 5109 cpu_manager.go:414] "containerMap: removing container" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466797 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466807 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466811 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466824 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" Nov 02 22:30:18.466897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466829 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466934 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466943 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e17753b-68c5-4a7b-a87b-cd0275a77b1f" containerName="istio-proxy" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466948 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" containerName="revoke-keys" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466953 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7802e21-b667-4a85-8e00-78b67e4a2548" containerName="istio-proxy" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466959 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466964 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerName="cleanup" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466971 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4ec82f-1f47-449f-8c45-244460554d96" containerName="istio-proxy" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466978 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466985 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7539982-c036-4526-9d1f-bc9324bed23f" containerName="istio-proxy" Nov 02 22:30:18.467229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.466991 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerName="cleanup" Nov 02 22:30:18.470861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.470844 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.614999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.614717 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2663d62e-gw-openshift-default-dockercfg-gp6w6\"" Nov 02 22:30:18.626913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.615190 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgh2t\" (UniqueName: \"kubernetes.io/projected/ed4edae0-0933-4c33-a0c9-fdf942973660-kube-api-access-wgh2t\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:18.697728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.691098 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8"] Nov 02 22:30:18.754592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.729818 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467202fa-5d84-41d6-b713-390961ca40f8-istiod-ca-cert\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.764118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.764087 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467202fa-5d84-41d6-b713-390961ca40f8-istio-podinfo\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.764524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.764497 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-istio-token\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.811775 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.811734 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z852k\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-kube-api-access-z852k\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.819291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.819242 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-socket\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.837282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.823791 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-data\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.837282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.823912 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-certs\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.867830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.827807 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-envoy\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.867830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.864456 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-credential-socket\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.961792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.961666 5109 scope.go:122] "RemoveContainer" containerID="19acf0e2bb33cefbfbeda679007762394766dba5b552b19d6c1b0938d866ae6d" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967121 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467202fa-5d84-41d6-b713-390961ca40f8-istiod-ca-cert\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967279 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467202fa-5d84-41d6-b713-390961ca40f8-istio-podinfo\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967406 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-istio-token\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967465 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z852k\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-kube-api-access-z852k\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967497 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-socket\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967580 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-data\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967612 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-certs\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967668 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-envoy\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.967695 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-credential-socket\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:18.971704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.968388 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-credential-socket\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.006461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.981271 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" event={"ID":"ed4edae0-0933-4c33-a0c9-fdf942973660","Type":"ContainerDied","Data":"f961e7c5c8beb6c2d8a25ec0dad7fcaf14d74179acb4b568af866037309c1395"} Nov 02 22:30:19.006461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.981351 5109 scope.go:122] "RemoveContainer" containerID="cd714cbfd8291d3ba7bced7d5efd4e4f18d40d4336c5197c6f8eae74a96eb382" Nov 02 22:30:19.006461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.981465 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h" Nov 02 22:30:19.006461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:18.991071 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467202fa-5d84-41d6-b713-390961ca40f8-istiod-ca-cert\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.021518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.012085 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-data\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.021518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.012980 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-socket\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.021518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.016583 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-envoy\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.021518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.017131 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-certs\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.146636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.135019 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z852k\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-kube-api-access-z852k\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.155481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.155443 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467202fa-5d84-41d6-b713-390961ca40f8-istio-podinfo\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.161638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.160222 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-istio-token\") pod \"e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.188581 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.188525 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:19.492998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.474676 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h"] Nov 02 22:30:19.513956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:19.513914 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3224ea-29894310-wbw2h"] Nov 02 22:30:20.059835 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.034912 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" path="/var/lib/kubelet/pods/ed4edae0-0933-4c33-a0c9-fdf942973660/volumes" Nov 02 22:30:20.122461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.122418 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:30:20.127540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.127500 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:30:20.127787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.127770 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:30:20.140286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.133199 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8"] Nov 02 22:30:20.192121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.186161 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerStarted","Data":"9cda51f5f74ffe5126bce48c85ebb8eb528404b02b9722c680536cbbc31cc963"} Nov 02 22:30:20.198916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.198834 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:20.199941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.199877 5109 generic.go:356] "Generic (PLEG): container finished" podID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerID="cd86b9c0a0d6b617aeec15bb9da736f5668a5b1e43e99ec57532be2acf11948c" exitCode=60 Nov 02 22:30:20.200749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.200650 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" event={"ID":"94d2531f-0293-4eaa-b52d-41e23d5cf3b7","Type":"ContainerDied","Data":"cd86b9c0a0d6b617aeec15bb9da736f5668a5b1e43e99ec57532be2acf11948c"} Nov 02 22:30:20.205020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.201095 5109 scope.go:122] "RemoveContainer" containerID="3821d0fe1198040125bd347a1aa0cad0ecd6e0414d325d925b3fad40adba058f" Nov 02 22:30:20.205020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.204083 5109 scope.go:122] "RemoveContainer" containerID="cd86b9c0a0d6b617aeec15bb9da736f5668a5b1e43e99ec57532be2acf11948c" Nov 02 22:30:20.205020 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:20.204626 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-pzh8j_odh-ai-gateway-infra(94d2531f-0293-4eaa-b52d-41e23d5cf3b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" Nov 02 22:30:20.230173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.228838 5109 generic.go:356] "Generic (PLEG): container finished" podID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerID="ff651fdd6412058ca1eaf087d30b96fecd74c7fdbb0291657e1da7039cae9745" exitCode=60 Nov 02 22:30:20.230173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.228912 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" event={"ID":"301bca8a-2d01-4259-a40d-5ec276afa08e","Type":"ContainerDied","Data":"ff651fdd6412058ca1eaf087d30b96fecd74c7fdbb0291657e1da7039cae9745"} Nov 02 22:30:20.230173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.229326 5109 scope.go:122] "RemoveContainer" containerID="ff651fdd6412058ca1eaf087d30b96fecd74c7fdbb0291657e1da7039cae9745" Nov 02 22:30:20.230173 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:20.229731 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj_odh-ai-gateway-infra(301bca8a-2d01-4259-a40d-5ec276afa08e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" Nov 02 22:30:20.280060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.268292 5109 scope.go:122] "RemoveContainer" containerID="afefb7662ff136fdbf07f0566164498e860221178829fd190839b29a47dc1091" Nov 02 22:30:20.322690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.322648 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06158047-c5ef-4601-a2eb-1f81c543870c-kube-api-access-gg5s2\" (UniqueName: \"kubernetes.io/projected/06158047-c5ef-4601-a2eb-1f81c543870c-kube-api-access-gg5s2\") pod \"06158047-c5ef-4601-a2eb-1f81c543870c\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " Nov 02 22:30:20.323251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.323149 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06158047-c5ef-4601-a2eb-1f81c543870c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06158047-c5ef-4601-a2eb-1f81c543870c-maas-api-service-ca\") pod \"06158047-c5ef-4601-a2eb-1f81c543870c\" (UID: \"06158047-c5ef-4601-a2eb-1f81c543870c\") " Nov 02 22:30:20.327100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.326593 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06158047-c5ef-4601-a2eb-1f81c543870c-maas-api-service-ca" pod "06158047-c5ef-4601-a2eb-1f81c543870c" (UID: "06158047-c5ef-4601-a2eb-1f81c543870c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:20.342660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.328792 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06158047-c5ef-4601-a2eb-1f81c543870c-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:20.342660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.339536 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06158047-c5ef-4601-a2eb-1f81c543870c-kube-api-access-gg5s2" pod "06158047-c5ef-4601-a2eb-1f81c543870c" (UID: "06158047-c5ef-4601-a2eb-1f81c543870c"). InnerVolumeSpecName "kube-api-access-gg5s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:20.442032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:20.441995 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5s2\" (UniqueName: \"kubernetes.io/projected/06158047-c5ef-4601-a2eb-1f81c543870c-kube-api-access-gg5s2\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:21.244119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.244077 5109 generic.go:356] "Generic (PLEG): container finished" podID="08742801-a801-461e-b6da-c0c148426c77" containerID="9cda51f5f74ffe5126bce48c85ebb8eb528404b02b9722c680536cbbc31cc963" exitCode=60 Nov 02 22:30:21.244639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.244130 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerDied","Data":"9cda51f5f74ffe5126bce48c85ebb8eb528404b02b9722c680536cbbc31cc963"} Nov 02 22:30:21.244639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.244209 5109 scope.go:122] "RemoveContainer" containerID="19acf0e2bb33cefbfbeda679007762394766dba5b552b19d6c1b0938d866ae6d" Nov 02 22:30:21.244743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.244699 5109 scope.go:122] "RemoveContainer" containerID="9cda51f5f74ffe5126bce48c85ebb8eb528404b02b9722c680536cbbc31cc963" Nov 02 22:30:21.245164 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:21.245138 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9_odh-ai-gateway-infra(08742801-a801-461e-b6da-c0c148426c77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" podUID="08742801-a801-461e-b6da-c0c148426c77" Nov 02 22:30:21.248948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.248932 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" Nov 02 22:30:21.248948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.248936 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x" event={"ID":"06158047-c5ef-4601-a2eb-1f81c543870c","Type":"ContainerDied","Data":"d557e5071e5465dd0c563a3dc5f62c403d0d4b70c4abef46654296c7312ceb8c"} Nov 02 22:30:21.250736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.250714 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" event={"ID":"467202fa-5d84-41d6-b713-390961ca40f8","Type":"ContainerStarted","Data":"ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9"} Nov 02 22:30:21.250835 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.250763 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" event={"ID":"467202fa-5d84-41d6-b713-390961ca40f8","Type":"ContainerStarted","Data":"26e4043beb03427148bae59f57aba88067f485ac9fa42ba4ad6cc4b368361295"} Nov 02 22:30:21.260691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.260674 5109 scope.go:122] "RemoveContainer" containerID="dfa8e334293ef863a4a3936f867d240fd099c078ad811eb99a83b83cab1d6509" Nov 02 22:30:21.335086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.335046 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9"] Nov 02 22:30:21.454690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.454654 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj"] Nov 02 22:30:21.493332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.493281 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podStartSLOduration=3.493267606 podStartE2EDuration="3.493267606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:21.492614709 +0000 UTC m=+1457.757607929" watchObservedRunningTime="2026-11-02 22:30:21.493267606 +0000 UTC m=+1457.758260824" Nov 02 22:30:21.610978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.610465 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x"] Nov 02 22:30:21.634519 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.634477 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98a222-29894310-mgb6x"] Nov 02 22:30:21.648272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.648243 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:21.691080 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.691045 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j"] Nov 02 22:30:21.762509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.762440 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/301bca8a-2d01-4259-a40d-5ec276afa08e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301bca8a-2d01-4259-a40d-5ec276afa08e-maas-api-service-ca\") pod \"301bca8a-2d01-4259-a40d-5ec276afa08e\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " Nov 02 22:30:21.763088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.762809 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301bca8a-2d01-4259-a40d-5ec276afa08e-kube-api-access-zm68w\" (UniqueName: \"kubernetes.io/projected/301bca8a-2d01-4259-a40d-5ec276afa08e-kube-api-access-zm68w\") pod \"301bca8a-2d01-4259-a40d-5ec276afa08e\" (UID: \"301bca8a-2d01-4259-a40d-5ec276afa08e\") " Nov 02 22:30:21.764317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.764208 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301bca8a-2d01-4259-a40d-5ec276afa08e-maas-api-service-ca" pod "301bca8a-2d01-4259-a40d-5ec276afa08e" (UID: "301bca8a-2d01-4259-a40d-5ec276afa08e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:21.773852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.773776 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301bca8a-2d01-4259-a40d-5ec276afa08e-kube-api-access-zm68w" pod "301bca8a-2d01-4259-a40d-5ec276afa08e" (UID: "301bca8a-2d01-4259-a40d-5ec276afa08e"). InnerVolumeSpecName "kube-api-access-zm68w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:21.950597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.950539 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm68w\" (UniqueName: \"kubernetes.io/projected/301bca8a-2d01-4259-a40d-5ec276afa08e-kube-api-access-zm68w\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:21.950829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:21.950807 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301bca8a-2d01-4259-a40d-5ec276afa08e-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.025712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.025218 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" path="/var/lib/kubelet/pods/06158047-c5ef-4601-a2eb-1f81c543870c/volumes" Nov 02 22:30:22.051766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.051732 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:22.261051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.196886 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:22.261051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.203269 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-kube-api-access-92nwt\" (UniqueName: \"kubernetes.io/projected/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-kube-api-access-92nwt\") pod \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " Nov 02 22:30:22.261051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.203458 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-maas-api-service-ca\") pod \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\" (UID: \"94d2531f-0293-4eaa-b52d-41e23d5cf3b7\") " Nov 02 22:30:22.261051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.259935 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-maas-api-service-ca" pod "94d2531f-0293-4eaa-b52d-41e23d5cf3b7" (UID: "94d2531f-0293-4eaa-b52d-41e23d5cf3b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:22.285858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.284950 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:22.329962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.312660 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.371613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.370328 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-kube-api-access-92nwt" pod "94d2531f-0293-4eaa-b52d-41e23d5cf3b7" (UID: "94d2531f-0293-4eaa-b52d-41e23d5cf3b7"). InnerVolumeSpecName "kube-api-access-92nwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:22.499789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.499695 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nwt\" (UniqueName: \"kubernetes.io/projected/94d2531f-0293-4eaa-b52d-41e23d5cf3b7-kube-api-access-92nwt\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.516652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.513331 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" event={"ID":"301bca8a-2d01-4259-a40d-5ec276afa08e","Type":"ContainerDied","Data":"afcd1c99b47028222cc482c2ca8432bd12aa44ddc34686b228848b28cddadc9d"} Nov 02 22:30:22.516652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.513407 5109 scope.go:122] "RemoveContainer" containerID="ff651fdd6412058ca1eaf087d30b96fecd74c7fdbb0291657e1da7039cae9745" Nov 02 22:30:22.516652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.513587 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj" Nov 02 22:30:22.536645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.536454 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" event={"ID":"94d2531f-0293-4eaa-b52d-41e23d5cf3b7","Type":"ContainerDied","Data":"ba3caf7c7ac57690955e3bbb7d91634edcd4cd4e68c0a1b6e42c8d9e18ca201f"} Nov 02 22:30:22.537280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.537069 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j" Nov 02 22:30:22.537280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.537225 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:22.539059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.539004 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:30:22.551984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.551955 5109 scope.go:122] "RemoveContainer" containerID="cd86b9c0a0d6b617aeec15bb9da736f5668a5b1e43e99ec57532be2acf11948c" Nov 02 22:30:22.905472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.905369 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j"] Nov 02 22:30:22.911029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.910932 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:22.958593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:22.936935 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-pzh8j"] Nov 02 22:30:23.008080 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.008002 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj"] Nov 02 22:30:23.013188 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.013126 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca85db-29894310-7mpsj"] Nov 02 22:30:23.054216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.054163 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08742801-a801-461e-b6da-c0c148426c77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08742801-a801-461e-b6da-c0c148426c77-maas-api-service-ca\") pod \"08742801-a801-461e-b6da-c0c148426c77\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " Nov 02 22:30:23.054559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.054540 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08742801-a801-461e-b6da-c0c148426c77-kube-api-access-sp9tm\" (UniqueName: \"kubernetes.io/projected/08742801-a801-461e-b6da-c0c148426c77-kube-api-access-sp9tm\") pod \"08742801-a801-461e-b6da-c0c148426c77\" (UID: \"08742801-a801-461e-b6da-c0c148426c77\") " Nov 02 22:30:23.054866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.054683 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08742801-a801-461e-b6da-c0c148426c77-maas-api-service-ca" pod "08742801-a801-461e-b6da-c0c148426c77" (UID: "08742801-a801-461e-b6da-c0c148426c77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:23.058252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.055997 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08742801-a801-461e-b6da-c0c148426c77-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:23.058252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.057191 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08742801-a801-461e-b6da-c0c148426c77-kube-api-access-sp9tm" pod "08742801-a801-461e-b6da-c0c148426c77" (UID: "08742801-a801-461e-b6da-c0c148426c77"). InnerVolumeSpecName "kube-api-access-sp9tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:23.159125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.159021 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp9tm\" (UniqueName: \"kubernetes.io/projected/08742801-a801-461e-b6da-c0c148426c77-kube-api-access-sp9tm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:23.251073 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.251027 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" podUID="c145e338-1ae0-4d01-889d-f2f48ae7ef16" containerName="payload-processing" containerID="cri-o://96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c" gracePeriod=25 Nov 02 22:30:23.532541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.532514 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:30:23.548707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.548674 5109 generic.go:356] "Generic (PLEG): container finished" podID="c145e338-1ae0-4d01-889d-f2f48ae7ef16" containerID="96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c" exitCode=0 Nov 02 22:30:23.548892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.548740 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" Nov 02 22:30:23.548892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.548768 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" event={"ID":"c145e338-1ae0-4d01-889d-f2f48ae7ef16","Type":"ContainerDied","Data":"96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c"} Nov 02 22:30:23.548892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.548831 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc" event={"ID":"c145e338-1ae0-4d01-889d-f2f48ae7ef16","Type":"ContainerDied","Data":"c0cd24a4e58c395b127dd8717f5d168efe918fbf1459d10ab57e5059fde534e0"} Nov 02 22:30:23.548892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.548862 5109 scope.go:122] "RemoveContainer" containerID="96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c" Nov 02 22:30:23.552495 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.552468 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" Nov 02 22:30:23.552690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.552472 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9" event={"ID":"08742801-a801-461e-b6da-c0c148426c77","Type":"ContainerDied","Data":"db4ecf38ab740a510866dfe292339d95dbbe8010f600eab9bc0067a76c992160"} Nov 02 22:30:23.562388 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.562369 5109 scope.go:122] "RemoveContainer" containerID="96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c" Nov 02 22:30:23.562748 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:23.562731 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c\": container with ID starting with 96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c not found: ID does not exist" containerID="96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c" Nov 02 22:30:23.562797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.562760 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c"} err="failed to get container status \"96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c\": rpc error: code = NotFound desc = could not find container \"96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c\": container with ID starting with 96ab4da5507187d2876500990f54642533865557cd81de0a6f5d25435b699e6c not found: ID does not exist" Nov 02 22:30:23.562797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.562778 5109 scope.go:122] "RemoveContainer" containerID="9cda51f5f74ffe5126bce48c85ebb8eb528404b02b9722c680536cbbc31cc963" Nov 02 22:30:23.667238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.667199 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c145e338-1ae0-4d01-889d-f2f48ae7ef16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c145e338-1ae0-4d01-889d-f2f48ae7ef16-plugins-config-volume\") pod \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " Nov 02 22:30:23.667483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.667463 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c145e338-1ae0-4d01-889d-f2f48ae7ef16-kube-api-access-cfbw8\" (UniqueName: \"kubernetes.io/projected/c145e338-1ae0-4d01-889d-f2f48ae7ef16-kube-api-access-cfbw8\") pod \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\" (UID: \"c145e338-1ae0-4d01-889d-f2f48ae7ef16\") " Nov 02 22:30:23.668048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.668019 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c145e338-1ae0-4d01-889d-f2f48ae7ef16-plugins-config-volume" pod "c145e338-1ae0-4d01-889d-f2f48ae7ef16" (UID: "c145e338-1ae0-4d01-889d-f2f48ae7ef16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:23.668837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.668508 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c145e338-1ae0-4d01-889d-f2f48ae7ef16-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:23.670169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.670138 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c145e338-1ae0-4d01-889d-f2f48ae7ef16-kube-api-access-cfbw8" pod "c145e338-1ae0-4d01-889d-f2f48ae7ef16" (UID: "c145e338-1ae0-4d01-889d-f2f48ae7ef16"). InnerVolumeSpecName "kube-api-access-cfbw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:23.770033 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.769995 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfbw8\" (UniqueName: \"kubernetes.io/projected/c145e338-1ae0-4d01-889d-f2f48ae7ef16-kube-api-access-cfbw8\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:23.823706 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.823666 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9"] Nov 02 22:30:23.829068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.829033 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-49a557-29894310-w9hm9"] Nov 02 22:30:23.896124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.896088 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08742801-a801-461e-b6da-c0c148426c77" path="/var/lib/kubelet/pods/08742801-a801-461e-b6da-c0c148426c77/volumes" Nov 02 22:30:23.896719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.896702 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" path="/var/lib/kubelet/pods/301bca8a-2d01-4259-a40d-5ec276afa08e/volumes" Nov 02 22:30:23.897116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.897103 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" path="/var/lib/kubelet/pods/94d2531f-0293-4eaa-b52d-41e23d5cf3b7/volumes" Nov 02 22:30:23.974496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.974457 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc"] Nov 02 22:30:23.987054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:23.987013 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e86dfb1-6b76855d75-rcpnc"] Nov 02 22:30:24.370668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:24.370633 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j"] Nov 02 22:30:24.423595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:24.423436 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db36b079-88m8j"] Nov 02 22:30:26.057603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:26.045836 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d2f28e-907e-42e0-a6b4-891dfdc73444" path="/var/lib/kubelet/pods/56d2f28e-907e-42e0-a6b4-891dfdc73444/volumes" Nov 02 22:30:26.077134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:26.077093 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c145e338-1ae0-4d01-889d-f2f48ae7ef16" path="/var/lib/kubelet/pods/c145e338-1ae0-4d01-889d-f2f48ae7ef16/volumes" Nov 02 22:30:26.085205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:26.078492 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj"] Nov 02 22:30:26.085744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:26.085700 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" containerID="cri-o://f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04" gracePeriod=30 Nov 02 22:30:26.514837 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:26.514800 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:30:27.517765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:27.517724 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:30:27.547684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:27.518488 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:30:31.325251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.325221 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:30:31.507546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.507515 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-certs\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.507769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.507640 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-credential-socket\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.507769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.507670 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-envoy\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.507769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.507723 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-kube-api-access-lh89t\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-kube-api-access-lh89t\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.507942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.507812 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-socket\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.507942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.507857 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-token\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-token\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.508046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.507952 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-data\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.508046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.508039 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istiod-ca-cert\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.508181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.508129 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-podinfo\") pod \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\" (UID: \"3fa74d8b-b126-487a-8ce9-84724a64e4b2\") " Nov 02 22:30:31.508537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.508504 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-socket" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.508704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.508517 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-data" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.509112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.509089 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istiod-ca-cert" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:31.509401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.509354 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.509517 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.509500 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.509869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.509845 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-certs" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.510074 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.510054 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-credential-socket" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.511624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.511152 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-podinfo" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:31.513081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.513047 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-token" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:31.515349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.515318 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-envoy" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.516013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.515991 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-kube-api-access-lh89t" pod "3fa74d8b-b126-487a-8ce9-84724a64e4b2" (UID: "3fa74d8b-b126-487a-8ce9-84724a64e4b2"). InnerVolumeSpecName "kube-api-access-lh89t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:31.611184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.611144 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.611184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.611174 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.611389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.611191 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.611389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.611205 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.611389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.611215 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.611389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.611224 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa74d8b-b126-487a-8ce9-84724a64e4b2-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.611389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.611233 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh89t\" (UniqueName: \"kubernetes.io/projected/3fa74d8b-b126-487a-8ce9-84724a64e4b2-kube-api-access-lh89t\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.886467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.886386 5109 generic.go:356] "Generic (PLEG): container finished" podID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerID="f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04" exitCode=0 Nov 02 22:30:31.886655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.886507 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" Nov 02 22:30:31.902826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.902748 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" event={"ID":"3fa74d8b-b126-487a-8ce9-84724a64e4b2","Type":"ContainerDied","Data":"f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04"} Nov 02 22:30:31.902923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.902864 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj" event={"ID":"3fa74d8b-b126-487a-8ce9-84724a64e4b2","Type":"ContainerDied","Data":"71824a3e3c5ba620e1a8d9c0d9b879a02e01676909b29a4ccd1462f20522635a"} Nov 02 22:30:31.902923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.902904 5109 scope.go:122] "RemoveContainer" containerID="f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04" Nov 02 22:30:31.915712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.915681 5109 scope.go:122] "RemoveContainer" containerID="473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570" Nov 02 22:30:31.927687 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.927657 5109 scope.go:122] "RemoveContainer" containerID="f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04" Nov 02 22:30:31.928008 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:31.927988 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04\": container with ID starting with f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04 not found: ID does not exist" containerID="f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04" Nov 02 22:30:31.928065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.928018 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04"} err="failed to get container status \"f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04\": rpc error: code = NotFound desc = could not find container \"f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04\": container with ID starting with f59c8a18960da85d3de3f51cb92a90e1b6f8ef53fc67f860122ac9ed40c88c04 not found: ID does not exist" Nov 02 22:30:31.928065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.928038 5109 scope.go:122] "RemoveContainer" containerID="473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570" Nov 02 22:30:31.928294 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:31.928276 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570\": container with ID starting with 473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570 not found: ID does not exist" containerID="473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570" Nov 02 22:30:31.928338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:31.928301 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570"} err="failed to get container status \"473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570\": rpc error: code = NotFound desc = could not find container \"473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570\": container with ID starting with 473963113ad218a31861b3cf747df23cbba5d1e9a1bccafab039e99d4bbf5570 not found: ID does not exist" Nov 02 22:30:32.060721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:32.060684 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj"] Nov 02 22:30:32.066128 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:32.066096 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-db36b079-openshift-default-7d58695dbc-h68cj"] Nov 02 22:30:33.894371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:33.894322 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" path="/var/lib/kubelet/pods/3fa74d8b-b126-487a-8ce9-84724a64e4b2/volumes" Nov 02 22:30:38.053888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:38.053852 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj"] Nov 02 22:30:38.108777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:38.108739 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b93686c3-972dj"] Nov 02 22:30:39.895666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:39.895626 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769661e5-9a2f-4e95-b3c2-ade0dfe3a547" path="/var/lib/kubelet/pods/769661e5-9a2f-4e95-b3c2-ade0dfe3a547/volumes" Nov 02 22:30:43.051720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:43.051682 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88"] Nov 02 22:30:43.101501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:43.101464 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7208d9-w7g88"] Nov 02 22:30:43.893796 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:43.893760 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218b52b1-048e-455a-b688-b8dd6a0fa2c8" path="/var/lib/kubelet/pods/218b52b1-048e-455a-b688-b8dd6a0fa2c8/volumes" Nov 02 22:30:46.342632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:46.342588 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g"] Nov 02 22:30:46.365900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:46.365862 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2e86dfb1-wbw5g"] Nov 02 22:30:47.894795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:47.894755 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7c288a-4177-4f3b-a975-d5cbbf7ffae3" path="/var/lib/kubelet/pods/9d7c288a-4177-4f3b-a975-d5cbbf7ffae3/volumes" Nov 02 22:30:48.051995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:48.051961 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d"] Nov 02 22:30:48.102131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:48.102097 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8704f258-lrl5d"] Nov 02 22:30:49.894480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:49.894444 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66635b2-e0f2-4c30-8652-7fe13d667d27" path="/var/lib/kubelet/pods/c66635b2-e0f2-4c30-8652-7fe13d667d27/volumes" Nov 02 22:30:54.740361 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.740322 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv"] Nov 02 22:30:54.741086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741069 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c145e338-1ae0-4d01-889d-f2f48ae7ef16" containerName="payload-processing" Nov 02 22:30:54.741133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741087 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c145e338-1ae0-4d01-889d-f2f48ae7ef16" containerName="payload-processing" Nov 02 22:30:54.741133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741100 5109 cpu_manager.go:414] "containerMap: removing container" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:30:54.741133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741105 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:30:54.741133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741116 5109 cpu_manager.go:414] "containerMap: removing container" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:30:54.741133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741122 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:30:54.741133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741133 5109 cpu_manager.go:414] "containerMap: removing container" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741139 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741151 5109 cpu_manager.go:414] "containerMap: removing container" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741155 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741160 5109 cpu_manager.go:414] "containerMap: removing container" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741165 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741177 5109 cpu_manager.go:414] "containerMap: removing container" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741181 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741193 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741198 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e1dcad-21e3-47b0-a91f-ca1a53f037f0" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741207 5109 cpu_manager.go:414] "containerMap: removing container" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741211 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741223 5109 cpu_manager.go:414] "containerMap: removing container" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741226 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741238 5109 cpu_manager.go:414] "containerMap: removing container" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741242 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741255 5109 cpu_manager.go:414] "containerMap: removing container" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:30:54.741337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741259 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741358 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741370 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741377 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741383 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741391 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741397 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741401 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741406 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741411 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741418 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c145e338-1ae0-4d01-889d-f2f48ae7ef16" containerName="payload-processing" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741423 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741429 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741435 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4edae0-0933-4c33-a0c9-fdf942973660" containerName="cleanup" Nov 02 22:30:54.741883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.741440 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:30:54.745556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.745534 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.802979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.802940 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv"] Nov 02 22:30:54.827744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.827695 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.827958 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.827751 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4c5\" (UniqueName: \"kubernetes.io/projected/0751d6d8-d378-4ad7-b34f-3452368f57ee-kube-api-access-7j4c5\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.828150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.828122 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.844027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.843986 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:30:54.844220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.844038 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:30:54.929806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.929766 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.930023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.929888 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.930023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.929911 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4c5\" (UniqueName: \"kubernetes.io/projected/0751d6d8-d378-4ad7-b34f-3452368f57ee-kube-api-access-7j4c5\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.930469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.930447 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.932644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.932623 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:54.991530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:54.991448 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4c5\" (UniqueName: \"kubernetes.io/projected/0751d6d8-d378-4ad7-b34f-3452368f57ee-kube-api-access-7j4c5\") pod \"maas-api-revoke-sub-229108f6-nmwtv\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:55.060274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:55.060237 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:55.438160 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:30:55.438128 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0751d6d8_d378_4ad7_b34f_3452368f57ee.slice/crio-dfd915eefa4bb641d10b2aef3f70b5e2340c149db6c579764d10e8adc799a567 WatchSource:0}: Error finding container dfd915eefa4bb641d10b2aef3f70b5e2340c149db6c579764d10e8adc799a567: Status 404 returned error can't find the container with id dfd915eefa4bb641d10b2aef3f70b5e2340c149db6c579764d10e8adc799a567 Nov 02 22:30:55.445329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:55.445299 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv"] Nov 02 22:30:56.247628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:56.247591 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" event={"ID":"0751d6d8-d378-4ad7-b34f-3452368f57ee","Type":"ContainerStarted","Data":"4aac5f378868c76703b3f5735e34488f9af8d8aeced5c7e5f8e2199d1903bfc0"} Nov 02 22:30:56.248060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:56.247656 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" event={"ID":"0751d6d8-d378-4ad7-b34f-3452368f57ee","Type":"ContainerStarted","Data":"dfd915eefa4bb641d10b2aef3f70b5e2340c149db6c579764d10e8adc799a567"} Nov 02 22:30:56.368130 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:56.368069 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" podStartSLOduration=2.368052972 podStartE2EDuration="2.368052972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:56.363682679 +0000 UTC m=+1492.628675899" watchObservedRunningTime="2026-11-02 22:30:56.368052972 +0000 UTC m=+1492.633046190" Nov 02 22:30:56.551138 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:30:56.551065 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:30:57.267631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:57.267546 5109 generic.go:356] "Generic (PLEG): container finished" podID="0751d6d8-d378-4ad7-b34f-3452368f57ee" containerID="4aac5f378868c76703b3f5735e34488f9af8d8aeced5c7e5f8e2199d1903bfc0" exitCode=0 Nov 02 22:30:57.268152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:57.267793 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" event={"ID":"0751d6d8-d378-4ad7-b34f-3452368f57ee","Type":"ContainerDied","Data":"4aac5f378868c76703b3f5735e34488f9af8d8aeced5c7e5f8e2199d1903bfc0"} Nov 02 22:30:58.456261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.456231 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:58.491258 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.491221 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-openshift-ca\") pod \"0751d6d8-d378-4ad7-b34f-3452368f57ee\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " Nov 02 22:30:58.491467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.491295 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0751d6d8-d378-4ad7-b34f-3452368f57ee-kube-api-access-7j4c5\" (UniqueName: \"kubernetes.io/projected/0751d6d8-d378-4ad7-b34f-3452368f57ee-kube-api-access-7j4c5\") pod \"0751d6d8-d378-4ad7-b34f-3452368f57ee\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " Nov 02 22:30:58.491538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.491523 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-xks-ca\") pod \"0751d6d8-d378-4ad7-b34f-3452368f57ee\" (UID: \"0751d6d8-d378-4ad7-b34f-3452368f57ee\") " Nov 02 22:30:58.491645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.491610 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-openshift-ca" pod "0751d6d8-d378-4ad7-b34f-3452368f57ee" (UID: "0751d6d8-d378-4ad7-b34f-3452368f57ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:58.492239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.492149 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:58.493684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.493655 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0751d6d8-d378-4ad7-b34f-3452368f57ee-kube-api-access-7j4c5" pod "0751d6d8-d378-4ad7-b34f-3452368f57ee" (UID: "0751d6d8-d378-4ad7-b34f-3452368f57ee"). InnerVolumeSpecName "kube-api-access-7j4c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:58.493778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.493742 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-xks-ca" pod "0751d6d8-d378-4ad7-b34f-3452368f57ee" (UID: "0751d6d8-d378-4ad7-b34f-3452368f57ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:58.594722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.594639 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0751d6d8-d378-4ad7-b34f-3452368f57ee-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:58.594722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:58.594678 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4c5\" (UniqueName: \"kubernetes.io/projected/0751d6d8-d378-4ad7-b34f-3452368f57ee-kube-api-access-7j4c5\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:59.295771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:59.295741 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" Nov 02 22:30:59.295967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:59.295780 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv" event={"ID":"0751d6d8-d378-4ad7-b34f-3452368f57ee","Type":"ContainerDied","Data":"dfd915eefa4bb641d10b2aef3f70b5e2340c149db6c579764d10e8adc799a567"} Nov 02 22:30:59.295967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:30:59.295834 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfd915eefa4bb641d10b2aef3f70b5e2340c149db6c579764d10e8adc799a567" Nov 02 22:31:00.051494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:00.051460 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv"] Nov 02 22:31:00.056716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:00.056686 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f60d85a-d4fjv"] Nov 02 22:31:01.894172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:01.894135 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1334fb99-90bf-4323-9bcf-409de877994c" path="/var/lib/kubelet/pods/1334fb99-90bf-4323-9bcf-409de877994c/volumes" Nov 02 22:31:03.052743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:03.052695 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g"] Nov 02 22:31:03.067637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:03.067595 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c1d5f1-88s8g"] Nov 02 22:31:03.894820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:03.894786 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec2f927-b2df-43fb-bac4-a30aeb20a3fa" path="/var/lib/kubelet/pods/3ec2f927-b2df-43fb-bac4-a30aeb20a3fa/volumes" Nov 02 22:31:06.963448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.963412 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn"] Nov 02 22:31:06.964166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964140 5109 cpu_manager.go:414] "containerMap: removing container" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:31:06.964208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964166 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:31:06.964208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964183 5109 cpu_manager.go:414] "containerMap: removing container" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:31:06.964208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964188 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d2531f-0293-4eaa-b52d-41e23d5cf3b7" containerName="cleanup" Nov 02 22:31:06.964208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964202 5109 cpu_manager.go:414] "containerMap: removing container" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" Nov 02 22:31:06.964208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964209 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa74d8b-b126-487a-8ce9-84724a64e4b2" containerName="istio-proxy" Nov 02 22:31:06.964369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964225 5109 cpu_manager.go:414] "containerMap: removing container" podUID="0751d6d8-d378-4ad7-b34f-3452368f57ee" containerName="revoke-keys" Nov 02 22:31:06.964369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964230 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0751d6d8-d378-4ad7-b34f-3452368f57ee" containerName="revoke-keys" Nov 02 22:31:06.964369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964249 5109 cpu_manager.go:414] "containerMap: removing container" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:31:06.964369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964253 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:31:06.964369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964266 5109 cpu_manager.go:414] "containerMap: removing container" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:31:06.964369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964271 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06158047-c5ef-4601-a2eb-1f81c543870c" containerName="cleanup" Nov 02 22:31:06.964547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964375 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="0751d6d8-d378-4ad7-b34f-3452368f57ee" containerName="revoke-keys" Nov 02 22:31:06.964547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964386 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="301bca8a-2d01-4259-a40d-5ec276afa08e" containerName="cleanup" Nov 02 22:31:06.964547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.964396 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="08742801-a801-461e-b6da-c0c148426c77" containerName="cleanup" Nov 02 22:31:06.969511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:06.969489 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.024086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.024046 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn"] Nov 02 22:31:07.066975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.066937 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2663d62e\"" Nov 02 22:31:07.068933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.067425 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2663d62e-dockercfg-7n5k5\"" Nov 02 22:31:07.200503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.200459 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f9affa-b826-4bfb-a70c-4401a3a4fa26-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.326733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.326693 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbd4\" (UniqueName: \"kubernetes.io/projected/66f9affa-b826-4bfb-a70c-4401a3a4fa26-kube-api-access-vwbd4\") pod \"payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.375678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.375633 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf"] Nov 02 22:31:07.394783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.385204 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.467241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.456868 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc7cc\" (UniqueName: \"kubernetes.io/projected/4f88dc55-b356-4640-b632-8c64cb9e3a42-kube-api-access-mc7cc\") pod \"payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.467241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.457003 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f88dc55-b356-4640-b632-8c64cb9e3a42-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.467241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.457182 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f9affa-b826-4bfb-a70c-4401a3a4fa26-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.467241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.457217 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbd4\" (UniqueName: \"kubernetes.io/projected/66f9affa-b826-4bfb-a70c-4401a3a4fa26-kube-api-access-vwbd4\") pod \"payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.467241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.458661 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f9affa-b826-4bfb-a70c-4401a3a4fa26-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.468068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.468035 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf"] Nov 02 22:31:07.518991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.518960 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbd4\" (UniqueName: \"kubernetes.io/projected/66f9affa-b826-4bfb-a70c-4401a3a4fa26-kube-api-access-vwbd4\") pod \"payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.558421 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.558385 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc7cc\" (UniqueName: \"kubernetes.io/projected/4f88dc55-b356-4640-b632-8c64cb9e3a42-kube-api-access-mc7cc\") pod \"payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.558643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.558468 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f88dc55-b356-4640-b632-8c64cb9e3a42-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.559171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.559152 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f88dc55-b356-4640-b632-8c64cb9e3a42-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.584909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.584828 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:07.620368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.620333 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc7cc\" (UniqueName: \"kubernetes.io/projected/4f88dc55-b356-4640-b632-8c64cb9e3a42-kube-api-access-mc7cc\") pod \"payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.762019 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.761977 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:07.969481 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:31:07.969445 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f9affa_b826_4bfb_a70c_4401a3a4fa26.slice/crio-abe7f2d85e766f1af2bcacb7df0883b6bf03466393b14cd6f34eb696dce0db97 WatchSource:0}: Error finding container abe7f2d85e766f1af2bcacb7df0883b6bf03466393b14cd6f34eb696dce0db97: Status 404 returned error can't find the container with id abe7f2d85e766f1af2bcacb7df0883b6bf03466393b14cd6f34eb696dce0db97 Nov 02 22:31:07.978968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:07.978935 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn"] Nov 02 22:31:08.055138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.055105 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs"] Nov 02 22:31:08.062208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.062180 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f32035-lmmhs"] Nov 02 22:31:08.141752 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:31:08.141725 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f88dc55_b356_4640_b632_8c64cb9e3a42.slice/crio-067e68bd4003cbdbc3e2e8df38df19b237a708f9dc60d0540f200078cbed1a98 WatchSource:0}: Error finding container 067e68bd4003cbdbc3e2e8df38df19b237a708f9dc60d0540f200078cbed1a98: Status 404 returned error can't find the container with id 067e68bd4003cbdbc3e2e8df38df19b237a708f9dc60d0540f200078cbed1a98 Nov 02 22:31:08.146241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.146220 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf"] Nov 02 22:31:08.465208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.465171 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" event={"ID":"66f9affa-b826-4bfb-a70c-4401a3a4fa26","Type":"ContainerStarted","Data":"04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626"} Nov 02 22:31:08.465432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.465237 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" event={"ID":"66f9affa-b826-4bfb-a70c-4401a3a4fa26","Type":"ContainerStarted","Data":"abe7f2d85e766f1af2bcacb7df0883b6bf03466393b14cd6f34eb696dce0db97"} Nov 02 22:31:08.465432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.465264 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:08.467211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.467184 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" event={"ID":"4f88dc55-b356-4640-b632-8c64cb9e3a42","Type":"ContainerStarted","Data":"081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec"} Nov 02 22:31:08.467332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.467236 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" event={"ID":"4f88dc55-b356-4640-b632-8c64cb9e3a42","Type":"ContainerStarted","Data":"067e68bd4003cbdbc3e2e8df38df19b237a708f9dc60d0540f200078cbed1a98"} Nov 02 22:31:08.467332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.467267 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:08.583336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.583283 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" podStartSLOduration=2.583268703 podStartE2EDuration="2.583268703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:08.581902702 +0000 UTC m=+1504.846895921" watchObservedRunningTime="2026-11-02 22:31:08.583268703 +0000 UTC m=+1504.848261921" Nov 02 22:31:08.765595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:08.760803 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" podStartSLOduration=1.760778782 podStartE2EDuration="1.760778782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:08.705157142 +0000 UTC m=+1504.970150361" watchObservedRunningTime="2026-11-02 22:31:08.760778782 +0000 UTC m=+1505.025772002" Nov 02 22:31:09.894449 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:09.894408 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f26f78-6284-4545-93bb-8b2233a8119a" path="/var/lib/kubelet/pods/f7f26f78-6284-4545-93bb-8b2233a8119a/volumes" Nov 02 22:31:11.082703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:11.068250 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c"] Nov 02 22:31:11.123837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:11.114455 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42cbaaf-k4w2c"] Nov 02 22:31:11.893856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:11.893824 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfe17d0-ba17-4dd4-9a79-8d573b292b53" path="/var/lib/kubelet/pods/4cfe17d0-ba17-4dd4-9a79-8d573b292b53/volumes" Nov 02 22:31:15.786837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:15.786797 5109 scope.go:122] "RemoveContainer" containerID="b664da216b4a4a4aa57645751cee8f4408692ac3927ad3b5990f67a7b8f07813" Nov 02 22:31:19.479166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:19.479135 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:31:19.479665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:19.479215 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:31:22.206600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.206092 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql"] Nov 02 22:31:22.241591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.241467 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.304699 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.304232 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql"] Nov 02 22:31:22.355593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.344763 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fb670683\"" Nov 02 22:31:22.355593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.344963 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fb670683-dockercfg-xxfkg\"" Nov 02 22:31:22.361616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.359535 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.361616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.359762 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9vm\" (UniqueName: \"kubernetes.io/projected/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-kube-api-access-6s9vm\") pod \"payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.379306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.366944 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h"] Nov 02 22:31:22.447151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.424022 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:22.479342 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.462380 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9vm\" (UniqueName: \"kubernetes.io/projected/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-kube-api-access-6s9vm\") pod \"payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.479342 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.462616 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.480070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.480036 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.550427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.550345 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9vm\" (UniqueName: \"kubernetes.io/projected/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-kube-api-access-6s9vm\") pod \"payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.572626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.563632 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h"] Nov 02 22:31:22.572626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.564888 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2addc7c0-75ea-468a-964d-b47cb6ed7303-plugins-config-volume\") pod \"payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:22.572626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.565030 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22hkh\" (UniqueName: \"kubernetes.io/projected/2addc7c0-75ea-468a-964d-b47cb6ed7303-kube-api-access-22hkh\") pod \"payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:22.670592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.660886 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:22.680718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.680542 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22hkh\" (UniqueName: \"kubernetes.io/projected/2addc7c0-75ea-468a-964d-b47cb6ed7303-kube-api-access-22hkh\") pod \"payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:22.681400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.681269 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2addc7c0-75ea-468a-964d-b47cb6ed7303-plugins-config-volume\") pod \"payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:22.682946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.682920 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2addc7c0-75ea-468a-964d-b47cb6ed7303-plugins-config-volume\") pod \"payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:22.838301 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.835991 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22hkh\" (UniqueName: \"kubernetes.io/projected/2addc7c0-75ea-468a-964d-b47cb6ed7303-kube-api-access-22hkh\") pod \"payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:22.859100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:22.859063 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:23.215205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.215170 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql"] Nov 02 22:31:23.257268 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:31:23.257135 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b93c72f_8c9b_4416_8d21_b8d2f894b19a.slice/crio-169cfa62c300e3983fb770ba79571a37be6e1d03564320ca266157a73aeb71f7 WatchSource:0}: Error finding container 169cfa62c300e3983fb770ba79571a37be6e1d03564320ca266157a73aeb71f7: Status 404 returned error can't find the container with id 169cfa62c300e3983fb770ba79571a37be6e1d03564320ca266157a73aeb71f7 Nov 02 22:31:23.262063 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:31:23.262029 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2addc7c0_75ea_468a_964d_b47cb6ed7303.slice/crio-6cd178cd6e38c03ec5adecc35e49afda61dfced2ab2fec94518c840f6ff649e8 WatchSource:0}: Error finding container 6cd178cd6e38c03ec5adecc35e49afda61dfced2ab2fec94518c840f6ff649e8: Status 404 returned error can't find the container with id 6cd178cd6e38c03ec5adecc35e49afda61dfced2ab2fec94518c840f6ff649e8 Nov 02 22:31:23.268099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.268074 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h"] Nov 02 22:31:23.862075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.862029 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" event={"ID":"0b93c72f-8c9b-4416-8d21-b8d2f894b19a","Type":"ContainerStarted","Data":"fe570f8b5d9f25e5cf067ae228dfcc733493be43b4c2dc5bf9150bffce9cab5c"} Nov 02 22:31:23.862284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.862090 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" event={"ID":"0b93c72f-8c9b-4416-8d21-b8d2f894b19a","Type":"ContainerStarted","Data":"169cfa62c300e3983fb770ba79571a37be6e1d03564320ca266157a73aeb71f7"} Nov 02 22:31:23.862284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.862115 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:23.863989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.863963 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" event={"ID":"2addc7c0-75ea-468a-964d-b47cb6ed7303","Type":"ContainerStarted","Data":"c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76"} Nov 02 22:31:23.864159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.864005 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" event={"ID":"2addc7c0-75ea-468a-964d-b47cb6ed7303","Type":"ContainerStarted","Data":"6cd178cd6e38c03ec5adecc35e49afda61dfced2ab2fec94518c840f6ff649e8"} Nov 02 22:31:23.864159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.864048 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:23.979961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:23.979907 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" podStartSLOduration=1.979890876 podStartE2EDuration="1.979890876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:23.977590208 +0000 UTC m=+1520.242583429" watchObservedRunningTime="2026-11-02 22:31:23.979890876 +0000 UTC m=+1520.244884095" Nov 02 22:31:24.116638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:24.112880 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" podStartSLOduration=2.112862595 podStartE2EDuration="2.112862595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:24.107355061 +0000 UTC m=+1520.372348282" watchObservedRunningTime="2026-11-02 22:31:24.112862595 +0000 UTC m=+1520.377855815" Nov 02 22:31:25.084532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:25.084487 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq"] Nov 02 22:31:25.101478 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:25.101429 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b946d-x6gmq"] Nov 02 22:31:25.894909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:25.894878 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb0b162-af08-45b2-b477-a77c3f3e89c8" path="/var/lib/kubelet/pods/8bb0b162-af08-45b2-b477-a77c3f3e89c8/volumes" Nov 02 22:31:26.198701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:26.198427 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m"] Nov 02 22:31:26.219887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:26.219851 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e11d791-kf28m"] Nov 02 22:31:26.494428 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:31:26.494332 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:31:26.957596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:26.953788 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn"] Nov 02 22:31:27.054591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.048226 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.150658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.134921 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn"] Nov 02 22:31:27.150658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.137974 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ldr\" (UniqueName: \"kubernetes.io/projected/86194a23-285d-4637-bc7d-9162c7aed560-kube-api-access-d6ldr\") pod \"payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.150658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.138212 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86194a23-285d-4637-bc7d-9162c7aed560-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.150658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.139259 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bb1a7ce7-dockercfg-wpv87\"" Nov 02 22:31:27.150658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.139343 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bb1a7ce7\"" Nov 02 22:31:27.302592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.246429 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ldr\" (UniqueName: \"kubernetes.io/projected/86194a23-285d-4637-bc7d-9162c7aed560-kube-api-access-d6ldr\") pod \"payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.302592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.246693 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86194a23-285d-4637-bc7d-9162c7aed560-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.302592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.269436 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86194a23-285d-4637-bc7d-9162c7aed560-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.351596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.351307 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ldr\" (UniqueName: \"kubernetes.io/projected/86194a23-285d-4637-bc7d-9162c7aed560-kube-api-access-d6ldr\") pod \"payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.485442 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.425843 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt"] Nov 02 22:31:27.510594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.509811 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt"] Nov 02 22:31:27.510594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.509978 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:27.770628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.770087 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmq4\" (UniqueName: \"kubernetes.io/projected/89548a9b-fd47-49cd-994b-b45fa6d76fea-kube-api-access-6dmq4\") pod \"payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:27.770628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.770237 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89548a9b-fd47-49cd-994b-b45fa6d76fea-plugins-config-volume\") pod \"payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:27.770628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.770494 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:27.871784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.871745 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmq4\" (UniqueName: \"kubernetes.io/projected/89548a9b-fd47-49cd-994b-b45fa6d76fea-kube-api-access-6dmq4\") pod \"payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:27.872125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.872101 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89548a9b-fd47-49cd-994b-b45fa6d76fea-plugins-config-volume\") pod \"payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:27.873176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.873076 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89548a9b-fd47-49cd-994b-b45fa6d76fea-plugins-config-volume\") pod \"payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:27.914367 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.913549 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c9fcdd-da7b-472f-a922-d18360ec9bd6" path="/var/lib/kubelet/pods/35c9fcdd-da7b-472f-a922-d18360ec9bd6/volumes" Nov 02 22:31:27.963094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.963062 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmq4\" (UniqueName: \"kubernetes.io/projected/89548a9b-fd47-49cd-994b-b45fa6d76fea-kube-api-access-6dmq4\") pod \"payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:27.987391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:27.987360 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:28.204624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:28.203927 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn"] Nov 02 22:31:28.205602 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:31:28.204821 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86194a23_285d_4637_bc7d_9162c7aed560.slice/crio-36e950e9a15a0a6770d29dafcabccc3f1ecc83f55333a47ebffcff794f7c73b0 WatchSource:0}: Error finding container 36e950e9a15a0a6770d29dafcabccc3f1ecc83f55333a47ebffcff794f7c73b0: Status 404 returned error can't find the container with id 36e950e9a15a0a6770d29dafcabccc3f1ecc83f55333a47ebffcff794f7c73b0 Nov 02 22:31:28.446800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:28.435402 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt"] Nov 02 22:31:29.085673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:29.085638 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" event={"ID":"86194a23-285d-4637-bc7d-9162c7aed560","Type":"ContainerStarted","Data":"6879940abd441664d5380beef99c2120e576453554d5c8617e3588bac03a5afa"} Nov 02 22:31:29.085988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:29.085969 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" event={"ID":"86194a23-285d-4637-bc7d-9162c7aed560","Type":"ContainerStarted","Data":"36e950e9a15a0a6770d29dafcabccc3f1ecc83f55333a47ebffcff794f7c73b0"} Nov 02 22:31:29.086075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:29.086066 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:29.087754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:29.087728 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" event={"ID":"89548a9b-fd47-49cd-994b-b45fa6d76fea","Type":"ContainerStarted","Data":"0c148b2a387a53a719f56c23a897ec03277ce89db2c8b6dc1491af138d918a6c"} Nov 02 22:31:29.087878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:29.087774 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" event={"ID":"89548a9b-fd47-49cd-994b-b45fa6d76fea","Type":"ContainerStarted","Data":"600d07f829650d6002eab7fc45ac3e6c20df3a3b443dfcc7b856355e360f4ebe"} Nov 02 22:31:29.087878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:29.087804 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:29.205264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:29.205008 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" podStartSLOduration=3.20499257 podStartE2EDuration="3.20499257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:29.204382431 +0000 UTC m=+1525.469375653" watchObservedRunningTime="2026-11-02 22:31:29.20499257 +0000 UTC m=+1525.469985789" Nov 02 22:31:34.145346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:34.145308 5109 generic.go:356] "Generic (PLEG): container finished" podID="467202fa-5d84-41d6-b713-390961ca40f8" containerID="ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9" exitCode=0 Nov 02 22:31:34.145834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:34.145384 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" event={"ID":"467202fa-5d84-41d6-b713-390961ca40f8","Type":"ContainerDied","Data":"ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9"} Nov 02 22:31:34.145915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:34.145896 5109 scope.go:122] "RemoveContainer" containerID="ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9" Nov 02 22:31:34.147527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:34.147496 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:31:34.147621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:34.147591 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:31:34.147667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:34.147625 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:31:34.324005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:34.323932 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" podStartSLOduration=7.323908176 podStartE2EDuration="7.323908176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:29.332741766 +0000 UTC m=+1525.597734988" watchObservedRunningTime="2026-11-02 22:31:34.323908176 +0000 UTC m=+1530.588901397" Nov 02 22:31:35.000690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.000650 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:31:35.000887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.000737 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:31:35.052246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.052212 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb"] Nov 02 22:31:35.101517 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.101476 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3ac0ac9-r69lb"] Nov 02 22:31:35.155746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.155696 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" event={"ID":"467202fa-5d84-41d6-b713-390961ca40f8","Type":"ContainerStarted","Data":"56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d"} Nov 02 22:31:35.190656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.190620 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:31:35.192063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.191872 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:31:35.192063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.191942 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 22:31:35.896383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:35.896347 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d219b9b9-b686-4d46-b944-87bb6f9e022b" path="/var/lib/kubelet/pods/d219b9b9-b686-4d46-b944-87bb6f9e022b/volumes" Nov 02 22:31:36.200842 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:36.200743 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:31:36.201460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:36.201418 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 22:31:37.055213 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:37.055172 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp"] Nov 02 22:31:37.099846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:37.099801 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a045f2-b7dzp"] Nov 02 22:31:37.191562 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:37.191519 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:31:37.191829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:37.191655 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 22:31:37.539729 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:37.539681 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:31:37.913511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:37.913386 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07398f0-55be-42f6-a86e-39d2f96f77fa" path="/var/lib/kubelet/pods/c07398f0-55be-42f6-a86e-39d2f96f77fa/volumes" Nov 02 22:31:38.190911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:38.190825 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:31:38.191085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:38.190907 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 22:31:39.055781 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.055742 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96"] Nov 02 22:31:39.103962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.103923 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq"] Nov 02 22:31:39.108692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.108667 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeecc3bd-whd96"] Nov 02 22:31:39.113370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.113339 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fc36ea-pzkmq"] Nov 02 22:31:39.197134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.197096 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:31:39.197424 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.197394 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 22:31:39.899804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.899766 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acce91e-d524-475d-8d70-b488fe67f151" path="/var/lib/kubelet/pods/1acce91e-d524-475d-8d70-b488fe67f151/volumes" Nov 02 22:31:39.901003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:39.900976 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b22b7b-c31f-4568-913a-095eae5a9c8d" path="/var/lib/kubelet/pods/70b22b7b-c31f-4568-913a-095eae5a9c8d/volumes" Nov 02 22:31:40.139894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:40.139834 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:31:40.146863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:40.146828 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:31:40.192142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:40.192045 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:31:40.192466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:40.192431 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 22:31:41.191077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:41.191039 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:31:41.191618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:41.191097 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 22:31:42.195757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:42.195723 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:31:42.196954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:42.196931 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:31:46.552450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.552411 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p"] Nov 02 22:31:46.557173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.557150 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.612616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.612580 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p"] Nov 02 22:31:46.637290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.637247 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.637481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.637301 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.637581 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.637540 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kczck\" (UniqueName: \"kubernetes.io/projected/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-kube-api-access-kczck\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.695490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.695457 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:46.696032 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.695959 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:31:46.739866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.739827 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.741086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.741058 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.741868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.740937 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.743298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.743275 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kczck\" (UniqueName: \"kubernetes.io/projected/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-kube-api-access-kczck\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.763420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.763381 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.819986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.819903 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kczck\" (UniqueName: \"kubernetes.io/projected/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-kube-api-access-kczck\") pod \"maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:46.872781 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:46.872744 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:47.252182 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:31:47.252140 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042c0e84_29e2_4ac2_b8af_45a6eb6ffb0a.slice/crio-34d70c266831054674a4fa9d2c4bae4d92190a69f1126c855ac3d36be77d459c WatchSource:0}: Error finding container 34d70c266831054674a4fa9d2c4bae4d92190a69f1126c855ac3d36be77d459c: Status 404 returned error can't find the container with id 34d70c266831054674a4fa9d2c4bae4d92190a69f1126c855ac3d36be77d459c Nov 02 22:31:47.256609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.256549 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p"] Nov 02 22:31:47.314931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.314899 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x"] Nov 02 22:31:47.319953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.319933 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.374751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.374716 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x"] Nov 02 22:31:47.456385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.456337 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.456636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.456411 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.456636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.456493 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2s5\" (UniqueName: \"kubernetes.io/projected/8b23b36d-9139-46ea-98fe-5fd739865b32-kube-api-access-ls2s5\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.558118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.558083 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.558502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.558152 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.558502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.558190 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2s5\" (UniqueName: \"kubernetes.io/projected/8b23b36d-9139-46ea-98fe-5fd739865b32-kube-api-access-ls2s5\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.558844 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.558825 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.560758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.560733 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.580035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.580000 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" event={"ID":"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a","Type":"ContainerStarted","Data":"d3bd07e868420ab2a99ff2bc1dc3afcc8c9918e1c8c1a687f7cbf14ad5eb1c97"} Nov 02 22:31:47.580217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.580064 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" event={"ID":"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a","Type":"ContainerStarted","Data":"34d70c266831054674a4fa9d2c4bae4d92190a69f1126c855ac3d36be77d459c"} Nov 02 22:31:47.619712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.619673 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2s5\" (UniqueName: \"kubernetes.io/projected/8b23b36d-9139-46ea-98fe-5fd739865b32-kube-api-access-ls2s5\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.634841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.634804 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:47.703550 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:47.703483 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" podStartSLOduration=1.703460516 podStartE2EDuration="1.703460516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:47.699512952 +0000 UTC m=+1543.964506172" watchObservedRunningTime="2026-11-02 22:31:47.703460516 +0000 UTC m=+1543.968453736" Nov 02 22:31:48.017933 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:31:48.017891 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b23b36d_9139_46ea_98fe_5fd739865b32.slice/crio-eabe8f4a4ce49b157a6934c7f2b131251640e7fe0cb7f3a130b7e04a3bd662f8 WatchSource:0}: Error finding container eabe8f4a4ce49b157a6934c7f2b131251640e7fe0cb7f3a130b7e04a3bd662f8: Status 404 returned error can't find the container with id eabe8f4a4ce49b157a6934c7f2b131251640e7fe0cb7f3a130b7e04a3bd662f8 Nov 02 22:31:48.022831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:48.022804 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x"] Nov 02 22:31:48.590925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:48.590893 5109 generic.go:356] "Generic (PLEG): container finished" podID="042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" containerID="d3bd07e868420ab2a99ff2bc1dc3afcc8c9918e1c8c1a687f7cbf14ad5eb1c97" exitCode=0 Nov 02 22:31:48.591387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:48.590984 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" event={"ID":"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a","Type":"ContainerDied","Data":"d3bd07e868420ab2a99ff2bc1dc3afcc8c9918e1c8c1a687f7cbf14ad5eb1c97"} Nov 02 22:31:48.592764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:48.592739 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" event={"ID":"8b23b36d-9139-46ea-98fe-5fd739865b32","Type":"ContainerStarted","Data":"a090c6c0b26843b3f459765af8a75818610dd370cbb6895faec5f82398e75a2f"} Nov 02 22:31:48.592855 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:48.592782 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" event={"ID":"8b23b36d-9139-46ea-98fe-5fd739865b32","Type":"ContainerStarted","Data":"eabe8f4a4ce49b157a6934c7f2b131251640e7fe0cb7f3a130b7e04a3bd662f8"} Nov 02 22:31:48.829682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:48.829620 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" podStartSLOduration=1.829601599 podStartE2EDuration="1.829601599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:48.824879134 +0000 UTC m=+1545.089872354" watchObservedRunningTime="2026-11-02 22:31:48.829601599 +0000 UTC m=+1545.094594818" Nov 02 22:31:49.603598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.603528 5109 generic.go:356] "Generic (PLEG): container finished" podID="8b23b36d-9139-46ea-98fe-5fd739865b32" containerID="a090c6c0b26843b3f459765af8a75818610dd370cbb6895faec5f82398e75a2f" exitCode=0 Nov 02 22:31:49.604102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.603610 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" event={"ID":"8b23b36d-9139-46ea-98fe-5fd739865b32","Type":"ContainerDied","Data":"a090c6c0b26843b3f459765af8a75818610dd370cbb6895faec5f82398e75a2f"} Nov 02 22:31:49.799592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.799549 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:49.895758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.895679 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-kube-api-access-kczck\" (UniqueName: \"kubernetes.io/projected/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-kube-api-access-kczck\") pod \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " Nov 02 22:31:49.895913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.895783 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-xks-ca\") pod \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " Nov 02 22:31:49.895913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.895889 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-openshift-ca\") pod \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\" (UID: \"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a\") " Nov 02 22:31:49.896240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.896212 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-openshift-ca" pod "042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" (UID: "042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:49.896643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.896622 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:49.897977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.897947 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-xks-ca" pod "042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" (UID: "042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:49.898064 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.897950 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-kube-api-access-kczck" pod "042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" (UID: "042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a"). InnerVolumeSpecName "kube-api-access-kczck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:49.997904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.997864 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:49.997904 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:49.997905 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kczck\" (UniqueName: \"kubernetes.io/projected/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a-kube-api-access-kczck\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:50.614467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.614433 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" Nov 02 22:31:50.614959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.614472 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p" event={"ID":"042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a","Type":"ContainerDied","Data":"34d70c266831054674a4fa9d2c4bae4d92190a69f1126c855ac3d36be77d459c"} Nov 02 22:31:50.614959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.614529 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34d70c266831054674a4fa9d2c4bae4d92190a69f1126c855ac3d36be77d459c" Nov 02 22:31:50.806623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.806594 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:50.912696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.912559 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-xks-ca\") pod \"8b23b36d-9139-46ea-98fe-5fd739865b32\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " Nov 02 22:31:50.912881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.912833 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-openshift-ca\") pod \"8b23b36d-9139-46ea-98fe-5fd739865b32\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " Nov 02 22:31:50.912959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.912942 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b23b36d-9139-46ea-98fe-5fd739865b32-kube-api-access-ls2s5\" (UniqueName: \"kubernetes.io/projected/8b23b36d-9139-46ea-98fe-5fd739865b32-kube-api-access-ls2s5\") pod \"8b23b36d-9139-46ea-98fe-5fd739865b32\" (UID: \"8b23b36d-9139-46ea-98fe-5fd739865b32\") " Nov 02 22:31:50.913195 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.913170 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-openshift-ca" pod "8b23b36d-9139-46ea-98fe-5fd739865b32" (UID: "8b23b36d-9139-46ea-98fe-5fd739865b32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:50.913661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.913640 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:50.915040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.915011 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-xks-ca" pod "8b23b36d-9139-46ea-98fe-5fd739865b32" (UID: "8b23b36d-9139-46ea-98fe-5fd739865b32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:50.915185 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.915164 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b23b36d-9139-46ea-98fe-5fd739865b32-kube-api-access-ls2s5" pod "8b23b36d-9139-46ea-98fe-5fd739865b32" (UID: "8b23b36d-9139-46ea-98fe-5fd739865b32"). InnerVolumeSpecName "kube-api-access-ls2s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:50.983692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.983652 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp"] Nov 02 22:31:50.988005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.987970 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x"] Nov 02 22:31:50.996476 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:50.996442 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn"] Nov 02 22:31:51.016045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:51.016016 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls2s5\" (UniqueName: \"kubernetes.io/projected/8b23b36d-9139-46ea-98fe-5fd739865b32-kube-api-access-ls2s5\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:51.016045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:51.016045 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b23b36d-9139-46ea-98fe-5fd739865b32-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:51.624687 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:51.624652 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" Nov 02 22:31:51.625144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:51.624660 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x" event={"ID":"8b23b36d-9139-46ea-98fe-5fd739865b32","Type":"ContainerDied","Data":"eabe8f4a4ce49b157a6934c7f2b131251640e7fe0cb7f3a130b7e04a3bd662f8"} Nov 02 22:31:51.625144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:51.624781 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eabe8f4a4ce49b157a6934c7f2b131251640e7fe0cb7f3a130b7e04a3bd662f8" Nov 02 22:31:51.791216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:51.791184 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv"] Nov 02 22:31:53.053983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:53.053947 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb"] Nov 02 22:31:53.103831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:53.103800 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq"] Nov 02 22:31:53.108579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:53.108542 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee26d52d-7s4zb"] Nov 02 22:31:53.113122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:53.113097 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2951dd58-rbldq"] Nov 02 22:31:53.895364 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:53.895327 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a66a81c-c59c-4fe6-8732-9915fdbc455e" path="/var/lib/kubelet/pods/3a66a81c-c59c-4fe6-8732-9915fdbc455e/volumes" Nov 02 22:31:53.895853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:53.895838 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d3d159-c7a3-45d7-9918-08a13ffc1ec2" path="/var/lib/kubelet/pods/a0d3d159-c7a3-45d7-9918-08a13ffc1ec2/volumes" Nov 02 22:31:56.050076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:56.050038 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f"] Nov 02 22:31:56.099626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:56.099554 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5ff70e-w8n8f"] Nov 02 22:31:56.493161 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:31:56.493066 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:31:57.053634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:57.053590 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2"] Nov 02 22:31:57.103114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:57.103071 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a95627-zn6j2"] Nov 02 22:31:57.895121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:57.895086 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229402a9-b633-4f80-bf48-ea87e221d362" path="/var/lib/kubelet/pods/229402a9-b633-4f80-bf48-ea87e221d362/volumes" Nov 02 22:31:57.895657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:57.895642 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2391038f-56ca-4493-b019-73a01f0ba73a" path="/var/lib/kubelet/pods/2391038f-56ca-4493-b019-73a01f0ba73a/volumes" Nov 02 22:31:58.171244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:58.171139 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9"] Nov 02 22:31:59.052681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:59.052641 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj"] Nov 02 22:31:59.058290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:59.058260 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp"] Nov 02 22:31:59.062986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:59.062961 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c5a20a-dtngj"] Nov 02 22:31:59.101881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:59.101842 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e44456b-8gwcp"] Nov 02 22:31:59.973326 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:59.973266 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15550cb1-ff59-4485-b325-d0016be7c921" path="/var/lib/kubelet/pods/15550cb1-ff59-4485-b325-d0016be7c921/volumes" Nov 02 22:31:59.976100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:31:59.976070 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18692684-72be-42f9-9704-9acf4bc32081" path="/var/lib/kubelet/pods/18692684-72be-42f9-9704-9acf4bc32081/volumes" Nov 02 22:32:01.399376 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.399329 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw"] Nov 02 22:32:01.400379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.400353 5109 cpu_manager.go:414] "containerMap: removing container" podUID="8b23b36d-9139-46ea-98fe-5fd739865b32" containerName="revoke-keys" Nov 02 22:32:01.400379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.400377 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b23b36d-9139-46ea-98fe-5fd739865b32" containerName="revoke-keys" Nov 02 22:32:01.400550 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.400416 5109 cpu_manager.go:414] "containerMap: removing container" podUID="042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" containerName="revoke-keys" Nov 02 22:32:01.400550 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.400424 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" containerName="revoke-keys" Nov 02 22:32:01.400707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.400693 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" containerName="revoke-keys" Nov 02 22:32:01.400761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.400727 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b23b36d-9139-46ea-98fe-5fd739865b32" containerName="revoke-keys" Nov 02 22:32:01.406012 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.405989 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.453392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.453346 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf"] Nov 02 22:32:01.458915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.458886 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.459925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.459901 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw"] Nov 02 22:32:01.495776 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.495741 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:32:01.495954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.495834 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:32:01.518881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.518835 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx"] Nov 02 22:32:01.524987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.524951 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf"] Nov 02 22:32:01.525224 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.525072 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.539924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.539889 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/92f3878b-4a2a-4689-9a7b-0455af40d585-kube-api-access-ct595\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.540098 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.539944 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.540186 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.540126 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.540186 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.540156 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qfh\" (UniqueName: \"kubernetes.io/projected/811c907d-0581-4d24-adff-5d43ebd3abdb-kube-api-access-89qfh\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.540305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.540190 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.540369 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.540337 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.569257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.569218 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx"] Nov 02 22:32:01.641932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.641899 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.642114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.641948 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qfh\" (UniqueName: \"kubernetes.io/projected/811c907d-0581-4d24-adff-5d43ebd3abdb-kube-api-access-89qfh\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.642114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.641987 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.642114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642013 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.642114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642040 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2xn\" (UniqueName: \"kubernetes.io/projected/54750126-b5a6-4de7-82a3-b32c71bc6f45-kube-api-access-2b2xn\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.642348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642156 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.642348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642289 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.642527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642506 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/92f3878b-4a2a-4689-9a7b-0455af40d585-kube-api-access-ct595\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.642625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642607 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.642884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642859 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.642967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.642941 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.644538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.644513 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.645124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.645102 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.703553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.703474 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qfh\" (UniqueName: \"kubernetes.io/projected/811c907d-0581-4d24-adff-5d43ebd3abdb-kube-api-access-89qfh\") pod \"maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.706349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.706314 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/92f3878b-4a2a-4689-9a7b-0455af40d585-kube-api-access-ct595\") pod \"maas-api-revoke-sub-f74292b7-ddtqw\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.721880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.721847 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:01.744129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.744095 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.744297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.744152 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2xn\" (UniqueName: \"kubernetes.io/projected/54750126-b5a6-4de7-82a3-b32c71bc6f45-kube-api-access-2b2xn\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.744297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.744215 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.744973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.744919 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.746829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.746805 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.774307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.774260 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:01.837797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.837758 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn"] Nov 02 22:32:01.839468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.839394 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2xn\" (UniqueName: \"kubernetes.io/projected/54750126-b5a6-4de7-82a3-b32c71bc6f45-kube-api-access-2b2xn\") pod \"maas-api-revoke-sub-0bedd8c4-498jx\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:01.846081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:01.845815 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:02.138099 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:32:02.130788 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f3878b_4a2a_4689_9a7b_0455af40d585.slice/crio-b0fdbb0931c0a659fb368d5bb3dbd1b339377e57b4cd4606c41d7542bc0b5262 WatchSource:0}: Error finding container b0fdbb0931c0a659fb368d5bb3dbd1b339377e57b4cd4606c41d7542bc0b5262: Status 404 returned error can't find the container with id b0fdbb0931c0a659fb368d5bb3dbd1b339377e57b4cd4606c41d7542bc0b5262 Nov 02 22:32:02.222676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.218444 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw"] Nov 02 22:32:02.407762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.407713 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx"] Nov 02 22:32:02.445673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.439885 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf"] Nov 02 22:32:02.979951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.979908 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" event={"ID":"92f3878b-4a2a-4689-9a7b-0455af40d585","Type":"ContainerStarted","Data":"9857895186e8c40345dbc0780a16b5b7074fcec96643fc852e843640685307d5"} Nov 02 22:32:02.980141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.979977 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" event={"ID":"92f3878b-4a2a-4689-9a7b-0455af40d585","Type":"ContainerStarted","Data":"b0fdbb0931c0a659fb368d5bb3dbd1b339377e57b4cd4606c41d7542bc0b5262"} Nov 02 22:32:02.981902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.981872 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" event={"ID":"54750126-b5a6-4de7-82a3-b32c71bc6f45","Type":"ContainerStarted","Data":"2ebd1a27a3ea037960ff5352ebbaa139dc83aa0da521af3a6d24949b8b4f07bd"} Nov 02 22:32:02.982068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.981923 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" event={"ID":"54750126-b5a6-4de7-82a3-b32c71bc6f45","Type":"ContainerStarted","Data":"348b1c5e2a2a03830fc4367ea09b749f56372485851e5839f29362e5abcc4f0b"} Nov 02 22:32:02.983670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.983638 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" event={"ID":"811c907d-0581-4d24-adff-5d43ebd3abdb","Type":"ContainerStarted","Data":"7a6d50c21bda1289f322d7b2e290e459cce1349d18248cb8522861690dbde457"} Nov 02 22:32:02.983791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:02.983680 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" event={"ID":"811c907d-0581-4d24-adff-5d43ebd3abdb","Type":"ContainerStarted","Data":"0fa77cca5eef4147154e685874ba15bc0875e7aacf2e941ce817d7a58b3f7b92"} Nov 02 22:32:03.110695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.110462 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" podStartSLOduration=2.110439015 podStartE2EDuration="2.110439015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:03.107689407 +0000 UTC m=+1559.372682618" watchObservedRunningTime="2026-11-02 22:32:03.110439015 +0000 UTC m=+1559.375432240" Nov 02 22:32:03.187875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.187827 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" podUID="d3a5d192-58f0-4d9f-8111-1308242009bd" containerName="payload-processing" containerID="cri-o://5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a" gracePeriod=25 Nov 02 22:32:03.266153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.266080 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" podStartSLOduration=2.266054027 podStartE2EDuration="2.266054027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:03.234032388 +0000 UTC m=+1559.499025609" watchObservedRunningTime="2026-11-02 22:32:03.266054027 +0000 UTC m=+1559.531047247" Nov 02 22:32:03.475894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.467634 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" podStartSLOduration=2.46760522 podStartE2EDuration="2.46760522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:03.455345026 +0000 UTC m=+1559.720338246" watchObservedRunningTime="2026-11-02 22:32:03.46760522 +0000 UTC m=+1559.732598438" Nov 02 22:32:03.890695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.890668 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:32:03.994983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.994946 5109 generic.go:356] "Generic (PLEG): container finished" podID="54750126-b5a6-4de7-82a3-b32c71bc6f45" containerID="2ebd1a27a3ea037960ff5352ebbaa139dc83aa0da521af3a6d24949b8b4f07bd" exitCode=0 Nov 02 22:32:03.995182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.995022 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" event={"ID":"54750126-b5a6-4de7-82a3-b32c71bc6f45","Type":"ContainerDied","Data":"2ebd1a27a3ea037960ff5352ebbaa139dc83aa0da521af3a6d24949b8b4f07bd"} Nov 02 22:32:03.996623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.996600 5109 generic.go:356] "Generic (PLEG): container finished" podID="811c907d-0581-4d24-adff-5d43ebd3abdb" containerID="7a6d50c21bda1289f322d7b2e290e459cce1349d18248cb8522861690dbde457" exitCode=0 Nov 02 22:32:03.996747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.996681 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" event={"ID":"811c907d-0581-4d24-adff-5d43ebd3abdb","Type":"ContainerDied","Data":"7a6d50c21bda1289f322d7b2e290e459cce1349d18248cb8522861690dbde457"} Nov 02 22:32:03.998353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.998334 5109 generic.go:356] "Generic (PLEG): container finished" podID="92f3878b-4a2a-4689-9a7b-0455af40d585" containerID="9857895186e8c40345dbc0780a16b5b7074fcec96643fc852e843640685307d5" exitCode=0 Nov 02 22:32:03.998449 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.998362 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" event={"ID":"92f3878b-4a2a-4689-9a7b-0455af40d585","Type":"ContainerDied","Data":"9857895186e8c40345dbc0780a16b5b7074fcec96643fc852e843640685307d5"} Nov 02 22:32:03.999754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.999735 5109 generic.go:356] "Generic (PLEG): container finished" podID="d3a5d192-58f0-4d9f-8111-1308242009bd" containerID="5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a" exitCode=0 Nov 02 22:32:03.999854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.999775 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" Nov 02 22:32:03.999854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.999812 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" event={"ID":"d3a5d192-58f0-4d9f-8111-1308242009bd","Type":"ContainerDied","Data":"5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a"} Nov 02 22:32:03.999854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.999850 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9" event={"ID":"d3a5d192-58f0-4d9f-8111-1308242009bd","Type":"ContainerDied","Data":"a75d141f1f59f42125d60f5a829232f82f90ac92d2ad6a46c7fd638e53cc4a4a"} Nov 02 22:32:03.999974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:03.999870 5109 scope.go:122] "RemoveContainer" containerID="5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a" Nov 02 22:32:04.014608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.014589 5109 scope.go:122] "RemoveContainer" containerID="5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a" Nov 02 22:32:04.014971 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:04.014951 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a\": container with ID starting with 5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a not found: ID does not exist" containerID="5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a" Nov 02 22:32:04.015038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.014980 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a"} err="failed to get container status \"5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a\": rpc error: code = NotFound desc = could not find container \"5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a\": container with ID starting with 5a151958a0aa19a2236cb246129b6c2cd8fe6a2a5f062d5ad9f7abad755f007a not found: ID does not exist" Nov 02 22:32:04.048226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.048188 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a5d192-58f0-4d9f-8111-1308242009bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d192-58f0-4d9f-8111-1308242009bd-plugins-config-volume\") pod \"d3a5d192-58f0-4d9f-8111-1308242009bd\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " Nov 02 22:32:04.048680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.048595 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a5d192-58f0-4d9f-8111-1308242009bd-kube-api-access-d566z\" (UniqueName: \"kubernetes.io/projected/d3a5d192-58f0-4d9f-8111-1308242009bd-kube-api-access-d566z\") pod \"d3a5d192-58f0-4d9f-8111-1308242009bd\" (UID: \"d3a5d192-58f0-4d9f-8111-1308242009bd\") " Nov 02 22:32:04.048840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.048703 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a5d192-58f0-4d9f-8111-1308242009bd-plugins-config-volume" pod "d3a5d192-58f0-4d9f-8111-1308242009bd" (UID: "d3a5d192-58f0-4d9f-8111-1308242009bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:04.049312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.049284 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d192-58f0-4d9f-8111-1308242009bd-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:04.053003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.051919 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a5d192-58f0-4d9f-8111-1308242009bd-kube-api-access-d566z" pod "d3a5d192-58f0-4d9f-8111-1308242009bd" (UID: "d3a5d192-58f0-4d9f-8111-1308242009bd"). InnerVolumeSpecName "kube-api-access-d566z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:04.053003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.052165 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2"] Nov 02 22:32:04.061875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.061848 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41fff3b-mg8v2"] Nov 02 22:32:04.150475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.150420 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d566z\" (UniqueName: \"kubernetes.io/projected/d3a5d192-58f0-4d9f-8111-1308242009bd-kube-api-access-d566z\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:04.525839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.525802 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9"] Nov 02 22:32:04.530290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:04.530264 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca85db-67548bb4ff-2sxb9"] Nov 02 22:32:05.198802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.198767 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:05.290335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.290309 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:05.293911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.293888 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:05.369149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.369110 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-openshift-ca\") pod \"54750126-b5a6-4de7-82a3-b32c71bc6f45\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " Nov 02 22:32:05.369350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.369207 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-xks-ca\") pod \"54750126-b5a6-4de7-82a3-b32c71bc6f45\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " Nov 02 22:32:05.369417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.369351 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54750126-b5a6-4de7-82a3-b32c71bc6f45-kube-api-access-2b2xn\" (UniqueName: \"kubernetes.io/projected/54750126-b5a6-4de7-82a3-b32c71bc6f45-kube-api-access-2b2xn\") pod \"54750126-b5a6-4de7-82a3-b32c71bc6f45\" (UID: \"54750126-b5a6-4de7-82a3-b32c71bc6f45\") " Nov 02 22:32:05.369496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.369462 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-openshift-ca" pod "54750126-b5a6-4de7-82a3-b32c71bc6f45" (UID: "54750126-b5a6-4de7-82a3-b32c71bc6f45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:05.370121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.370097 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.371617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.371557 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-xks-ca" pod "54750126-b5a6-4de7-82a3-b32c71bc6f45" (UID: "54750126-b5a6-4de7-82a3-b32c71bc6f45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:05.371776 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.371757 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54750126-b5a6-4de7-82a3-b32c71bc6f45-kube-api-access-2b2xn" pod "54750126-b5a6-4de7-82a3-b32c71bc6f45" (UID: "54750126-b5a6-4de7-82a3-b32c71bc6f45"). InnerVolumeSpecName "kube-api-access-2b2xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:05.471427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.471392 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-openshift-ca\") pod \"811c907d-0581-4d24-adff-5d43ebd3abdb\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " Nov 02 22:32:05.471688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.471602 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-xks-ca\") pod \"92f3878b-4a2a-4689-9a7b-0455af40d585\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " Nov 02 22:32:05.471801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.471778 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811c907d-0581-4d24-adff-5d43ebd3abdb-kube-api-access-89qfh\" (UniqueName: \"kubernetes.io/projected/811c907d-0581-4d24-adff-5d43ebd3abdb-kube-api-access-89qfh\") pod \"811c907d-0581-4d24-adff-5d43ebd3abdb\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " Nov 02 22:32:05.471868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.471819 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-xks-ca\") pod \"811c907d-0581-4d24-adff-5d43ebd3abdb\" (UID: \"811c907d-0581-4d24-adff-5d43ebd3abdb\") " Nov 02 22:32:05.471868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.471838 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-openshift-ca" pod "811c907d-0581-4d24-adff-5d43ebd3abdb" (UID: "811c907d-0581-4d24-adff-5d43ebd3abdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:05.471868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.471846 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-openshift-ca\") pod \"92f3878b-4a2a-4689-9a7b-0455af40d585\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " Nov 02 22:32:05.472030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.471904 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f3878b-4a2a-4689-9a7b-0455af40d585-kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/92f3878b-4a2a-4689-9a7b-0455af40d585-kube-api-access-ct595\") pod \"92f3878b-4a2a-4689-9a7b-0455af40d585\" (UID: \"92f3878b-4a2a-4689-9a7b-0455af40d585\") " Nov 02 22:32:05.472273 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.472249 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-openshift-ca" pod "92f3878b-4a2a-4689-9a7b-0455af40d585" (UID: "92f3878b-4a2a-4689-9a7b-0455af40d585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:05.473000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.472906 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.473100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.473006 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54750126-b5a6-4de7-82a3-b32c71bc6f45-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.473100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.473022 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.473100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.473039 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b2xn\" (UniqueName: \"kubernetes.io/projected/54750126-b5a6-4de7-82a3-b32c71bc6f45-kube-api-access-2b2xn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.474424 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.474389 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-xks-ca" pod "811c907d-0581-4d24-adff-5d43ebd3abdb" (UID: "811c907d-0581-4d24-adff-5d43ebd3abdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:05.474516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.474447 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-xks-ca" pod "92f3878b-4a2a-4689-9a7b-0455af40d585" (UID: "92f3878b-4a2a-4689-9a7b-0455af40d585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:05.474979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.474951 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811c907d-0581-4d24-adff-5d43ebd3abdb-kube-api-access-89qfh" pod "811c907d-0581-4d24-adff-5d43ebd3abdb" (UID: "811c907d-0581-4d24-adff-5d43ebd3abdb"). InnerVolumeSpecName "kube-api-access-89qfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:05.475134 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.475119 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f3878b-4a2a-4689-9a7b-0455af40d585-kube-api-access-ct595" pod "92f3878b-4a2a-4689-9a7b-0455af40d585" (UID: "92f3878b-4a2a-4689-9a7b-0455af40d585"). InnerVolumeSpecName "kube-api-access-ct595". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:05.575396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.575295 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3878b-4a2a-4689-9a7b-0455af40d585-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.575396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.575333 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qfh\" (UniqueName: \"kubernetes.io/projected/811c907d-0581-4d24-adff-5d43ebd3abdb-kube-api-access-89qfh\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.575396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.575351 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811c907d-0581-4d24-adff-5d43ebd3abdb-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.575396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.575366 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/92f3878b-4a2a-4689-9a7b-0455af40d585-kube-api-access-ct595\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:05.901513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.901412 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a5d192-58f0-4d9f-8111-1308242009bd" path="/var/lib/kubelet/pods/d3a5d192-58f0-4d9f-8111-1308242009bd/volumes" Nov 02 22:32:05.903872 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:05.902266 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99f5550-aa6f-4571-9b31-f1d964850aac" path="/var/lib/kubelet/pods/f99f5550-aa6f-4571-9b31-f1d964850aac/volumes" Nov 02 22:32:06.010454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.010381 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" containerID="cri-o://824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2" gracePeriod=45 Nov 02 22:32:06.011462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.011429 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" containerID="cri-o://57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815" gracePeriod=45 Nov 02 22:32:06.021875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.021851 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" Nov 02 22:32:06.021977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.021871 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw" event={"ID":"92f3878b-4a2a-4689-9a7b-0455af40d585","Type":"ContainerDied","Data":"b0fdbb0931c0a659fb368d5bb3dbd1b339377e57b4cd4606c41d7542bc0b5262"} Nov 02 22:32:06.021977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.021922 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0fdbb0931c0a659fb368d5bb3dbd1b339377e57b4cd4606c41d7542bc0b5262" Nov 02 22:32:06.023725 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.023694 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" containerID="cri-o://655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6" gracePeriod=45 Nov 02 22:32:06.023887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.023861 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" Nov 02 22:32:06.023957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.023884 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx" event={"ID":"54750126-b5a6-4de7-82a3-b32c71bc6f45","Type":"ContainerDied","Data":"348b1c5e2a2a03830fc4367ea09b749f56372485851e5839f29362e5abcc4f0b"} Nov 02 22:32:06.023957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.023940 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="348b1c5e2a2a03830fc4367ea09b749f56372485851e5839f29362e5abcc4f0b" Nov 02 22:32:06.026046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.026023 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" event={"ID":"811c907d-0581-4d24-adff-5d43ebd3abdb","Type":"ContainerDied","Data":"0fa77cca5eef4147154e685874ba15bc0875e7aacf2e941ce817d7a58b3f7b92"} Nov 02 22:32:06.026150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.026066 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa77cca5eef4147154e685874ba15bc0875e7aacf2e941ce817d7a58b3f7b92" Nov 02 22:32:06.026150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.026098 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf" Nov 02 22:32:06.052438 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.052405 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z"] Nov 02 22:32:06.080871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.080837 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e008e-55v7z"] Nov 02 22:32:06.370406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.370380 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:32:06.387851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.387693 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-home\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-home\") pod \"47c2f9cc-3ba0-4891-9790-d86df9dac639\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " Nov 02 22:32:06.387851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.387732 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c2f9cc-3ba0-4891-9790-d86df9dac639-kube-api-access-f6jr9\" (UniqueName: \"kubernetes.io/projected/47c2f9cc-3ba0-4891-9790-d86df9dac639-kube-api-access-f6jr9\") pod \"47c2f9cc-3ba0-4891-9790-d86df9dac639\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " Nov 02 22:32:06.387851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.387775 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-model-cache\") pod \"47c2f9cc-3ba0-4891-9790-d86df9dac639\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " Nov 02 22:32:06.388955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.388052 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-home" pod "47c2f9cc-3ba0-4891-9790-d86df9dac639" (UID: "47c2f9cc-3ba0-4891-9790-d86df9dac639"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.388955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.388209 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-model-cache" pod "47c2f9cc-3ba0-4891-9790-d86df9dac639" (UID: "47c2f9cc-3ba0-4891-9790-d86df9dac639"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.388955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.388708 5109 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-model-cache\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.388955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.388730 5109 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-home\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.390793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.390773 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:32:06.391095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.391070 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c2f9cc-3ba0-4891-9790-d86df9dac639-kube-api-access-f6jr9" pod "47c2f9cc-3ba0-4891-9790-d86df9dac639" (UID: "47c2f9cc-3ba0-4891-9790-d86df9dac639"). InnerVolumeSpecName "kube-api-access-f6jr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:06.396806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.396785 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:32:06.489465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489393 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c658010-2eae-42cf-8a70-f08f3014976c-kube-api-access-k4jfn\" (UniqueName: \"kubernetes.io/projected/8c658010-2eae-42cf-8a70-f08f3014976c-kube-api-access-k4jfn\") pod \"8c658010-2eae-42cf-8a70-f08f3014976c\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " Nov 02 22:32:06.489664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489476 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-dshm\") pod \"8c658010-2eae-42cf-8a70-f08f3014976c\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " Nov 02 22:32:06.489664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489511 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-model-cache\") pod \"8c658010-2eae-42cf-8a70-f08f3014976c\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " Nov 02 22:32:06.489664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489555 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-tmp-dir\") pod \"8c658010-2eae-42cf-8a70-f08f3014976c\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " Nov 02 22:32:06.489664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489604 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-dshm\") pod \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " Nov 02 22:32:06.489664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489635 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-kube-api-access-6qtl6\" (UniqueName: \"kubernetes.io/projected/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-kube-api-access-6qtl6\") pod \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " Nov 02 22:32:06.489959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489685 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tmp-dir\") pod \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " Nov 02 22:32:06.489959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489705 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-model-cache\") pod \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " Nov 02 22:32:06.489959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489800 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c2f9cc-3ba0-4891-9790-d86df9dac639-tls-certs\" (UniqueName: \"kubernetes.io/secret/47c2f9cc-3ba0-4891-9790-d86df9dac639-tls-certs\") pod \"47c2f9cc-3ba0-4891-9790-d86df9dac639\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " Nov 02 22:32:06.489959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489838 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tls-certs\") pod \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " Nov 02 22:32:06.489959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489858 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-tmp-dir\") pod \"47c2f9cc-3ba0-4891-9790-d86df9dac639\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " Nov 02 22:32:06.489959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489878 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-home\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-home\") pod \"8c658010-2eae-42cf-8a70-f08f3014976c\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " Nov 02 22:32:06.489959 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489948 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-dshm\") pod \"47c2f9cc-3ba0-4891-9790-d86df9dac639\" (UID: \"47c2f9cc-3ba0-4891-9790-d86df9dac639\") " Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.489973 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c658010-2eae-42cf-8a70-f08f3014976c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c658010-2eae-42cf-8a70-f08f3014976c-tls-certs\") pod \"8c658010-2eae-42cf-8a70-f08f3014976c\" (UID: \"8c658010-2eae-42cf-8a70-f08f3014976c\") " Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.490027 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-home\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-home\") pod \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\" (UID: \"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463\") " Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.490183 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tmp-dir" pod "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" (UID: "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.490391 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-model-cache" pod "8c658010-2eae-42cf-8a70-f08f3014976c" (UID: "8c658010-2eae-42cf-8a70-f08f3014976c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.490455 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-model-cache" pod "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" (UID: "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.490685 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-tmp-dir" pod "8c658010-2eae-42cf-8a70-f08f3014976c" (UID: "8c658010-2eae-42cf-8a70-f08f3014976c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.490689 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-home" pod "8c658010-2eae-42cf-8a70-f08f3014976c" (UID: "8c658010-2eae-42cf-8a70-f08f3014976c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.490710 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.490702 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-home" pod "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" (UID: "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.491421 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491397 5109 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-home\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.491505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491428 5109 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-model-cache\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.491505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491444 5109 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-tmp-dir\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.491505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491462 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6jr9\" (UniqueName: \"kubernetes.io/projected/47c2f9cc-3ba0-4891-9790-d86df9dac639-kube-api-access-f6jr9\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.491505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491479 5109 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tmp-dir\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.491505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491492 5109 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-model-cache\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.491791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491506 5109 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-home\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.492309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.491902 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-tmp-dir" pod "47c2f9cc-3ba0-4891-9790-d86df9dac639" (UID: "47c2f9cc-3ba0-4891-9790-d86df9dac639"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.492416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.492336 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-dshm" pod "8c658010-2eae-42cf-8a70-f08f3014976c" (UID: "8c658010-2eae-42cf-8a70-f08f3014976c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.492489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.492466 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c658010-2eae-42cf-8a70-f08f3014976c-kube-api-access-k4jfn" pod "8c658010-2eae-42cf-8a70-f08f3014976c" (UID: "8c658010-2eae-42cf-8a70-f08f3014976c"). InnerVolumeSpecName "kube-api-access-k4jfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:06.494689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.494666 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-dshm" pod "47c2f9cc-3ba0-4891-9790-d86df9dac639" (UID: "47c2f9cc-3ba0-4891-9790-d86df9dac639"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.514886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.502731 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c2f9cc-3ba0-4891-9790-d86df9dac639-tls-certs" pod "47c2f9cc-3ba0-4891-9790-d86df9dac639" (UID: "47c2f9cc-3ba0-4891-9790-d86df9dac639"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:06.514886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.502744 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-dshm" pod "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" (UID: "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:06.514886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.502729 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c658010-2eae-42cf-8a70-f08f3014976c-tls-certs" pod "8c658010-2eae-42cf-8a70-f08f3014976c" (UID: "8c658010-2eae-42cf-8a70-f08f3014976c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:06.514886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.502743 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tls-certs" pod "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" (UID: "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:06.514886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.502805 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-kube-api-access-6qtl6" pod "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" (UID: "9f43ab2f-5a4c-4759-9212-ae3ab7f7d463"). InnerVolumeSpecName "kube-api-access-6qtl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:06.593107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593071 5109 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c658010-2eae-42cf-8a70-f08f3014976c-dshm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593102 5109 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-dshm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593112 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qtl6\" (UniqueName: \"kubernetes.io/projected/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-kube-api-access-6qtl6\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593122 5109 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c2f9cc-3ba0-4891-9790-d86df9dac639-tls-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593131 5109 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463-tls-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593139 5109 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-tmp-dir\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593148 5109 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c2f9cc-3ba0-4891-9790-d86df9dac639-dshm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593155 5109 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c658010-2eae-42cf-8a70-f08f3014976c-tls-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.593558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.593163 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4jfn\" (UniqueName: \"kubernetes.io/projected/8c658010-2eae-42cf-8a70-f08f3014976c-kube-api-access-k4jfn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:06.815050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.815002 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" containerID="cri-o://58fa74e1945c4dac3161ab3d3f8e38a8e0457459c8cbe46e7400fe9922911af6" gracePeriod=45 Nov 02 22:32:06.838849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.838787 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" podUID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" containerName="payload-pre-processing" containerID="cri-o://8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71" gracePeriod=25 Nov 02 22:32:06.873912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:06.873876 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": dial tcp 10.133.0.84:8000: connect: connection refused" Nov 02 22:32:07.041712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.040988 5109 generic.go:356] "Generic (PLEG): container finished" podID="8c658010-2eae-42cf-8a70-f08f3014976c" containerID="824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2" exitCode=0 Nov 02 22:32:07.041712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.041043 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" Nov 02 22:32:07.041712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.041066 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" event={"ID":"8c658010-2eae-42cf-8a70-f08f3014976c","Type":"ContainerDied","Data":"824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2"} Nov 02 22:32:07.041712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.041114 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp" event={"ID":"8c658010-2eae-42cf-8a70-f08f3014976c","Type":"ContainerDied","Data":"8f1ecc54513ff4987125318d3650e88231976ed52a177972cfbc6e7f122790e7"} Nov 02 22:32:07.041712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.041177 5109 scope.go:122] "RemoveContainer" containerID="824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2" Nov 02 22:32:07.047459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.045111 5109 generic.go:356] "Generic (PLEG): container finished" podID="d59ef92f-e353-45e2-9512-70791c54d288" containerID="58fa74e1945c4dac3161ab3d3f8e38a8e0457459c8cbe46e7400fe9922911af6" exitCode=0 Nov 02 22:32:07.047459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.045212 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" event={"ID":"d59ef92f-e353-45e2-9512-70791c54d288","Type":"ContainerDied","Data":"58fa74e1945c4dac3161ab3d3f8e38a8e0457459c8cbe46e7400fe9922911af6"} Nov 02 22:32:07.047459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.047186 5109 generic.go:356] "Generic (PLEG): container finished" podID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerID="655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6" exitCode=0 Nov 02 22:32:07.047459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.047306 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" event={"ID":"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463","Type":"ContainerDied","Data":"655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6"} Nov 02 22:32:07.047459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.047340 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" event={"ID":"9f43ab2f-5a4c-4759-9212-ae3ab7f7d463","Type":"ContainerDied","Data":"9c16d9bf088965f60bf1f235d809e19c75f685051a193defcfd791886b55be29"} Nov 02 22:32:07.047459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.047421 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn" Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.326202 5109 scope.go:122] "RemoveContainer" containerID="824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2" Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:07.329133 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2\": container with ID starting with 824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2 not found: ID does not exist" containerID="824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2" Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.329192 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2"} err="failed to get container status \"824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2\": rpc error: code = NotFound desc = could not find container \"824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2\": container with ID starting with 824e354fcca9a0d17eafa6851a430a6f22b3c482fa020e28d871ea981a515bd2 not found: ID does not exist" Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.329231 5109 scope.go:122] "RemoveContainer" containerID="655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6" Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.332223 5109 generic.go:356] "Generic (PLEG): container finished" podID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerID="57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815" exitCode=0 Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.332269 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" event={"ID":"47c2f9cc-3ba0-4891-9790-d86df9dac639","Type":"ContainerDied","Data":"57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815"} Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.332314 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" event={"ID":"47c2f9cc-3ba0-4891-9790-d86df9dac639","Type":"ContainerDied","Data":"0da61f70c4d48cf60b0df99702f7808882252c74600ce09fe8c8d6404eceadf5"} Nov 02 22:32:07.336622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.332535 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x" Nov 02 22:32:07.452933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.452812 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:32:07.464981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.464956 5109 scope.go:122] "RemoveContainer" containerID="655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6" Nov 02 22:32:07.465369 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:07.465339 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6\": container with ID starting with 655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6 not found: ID does not exist" containerID="655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6" Nov 02 22:32:07.465451 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.465382 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6"} err="failed to get container status \"655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6\": rpc error: code = NotFound desc = could not find container \"655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6\": container with ID starting with 655cfb579dd89c7d1b3784721b007e0bcf40ce65d51647c888b7749f3f6dc2d6 not found: ID does not exist" Nov 02 22:32:07.465451 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.465409 5109 scope.go:122] "RemoveContainer" containerID="57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815" Nov 02 22:32:07.481008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.480978 5109 scope.go:122] "RemoveContainer" containerID="57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815" Nov 02 22:32:07.481436 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:07.481409 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815\": container with ID starting with 57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815 not found: ID does not exist" containerID="57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815" Nov 02 22:32:07.481529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.481448 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815"} err="failed to get container status \"57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815\": rpc error: code = NotFound desc = could not find container \"57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815\": container with ID starting with 57a7201035f4a96f0e368ccd365e209afd02d327158c75e3225da4fb2dcf5815 not found: ID does not exist" Nov 02 22:32:07.536270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536236 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-tmp-dir\") pod \"d59ef92f-e353-45e2-9512-70791c54d288\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " Nov 02 22:32:07.536270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536273 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-model-cache\") pod \"d59ef92f-e353-45e2-9512-70791c54d288\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " Nov 02 22:32:07.536535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536326 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-dshm\") pod \"d59ef92f-e353-45e2-9512-70791c54d288\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " Nov 02 22:32:07.536535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536356 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-home\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-home\") pod \"d59ef92f-e353-45e2-9512-70791c54d288\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " Nov 02 22:32:07.536535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536439 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59ef92f-e353-45e2-9512-70791c54d288-kube-api-access-v2lh9\" (UniqueName: \"kubernetes.io/projected/d59ef92f-e353-45e2-9512-70791c54d288-kube-api-access-v2lh9\") pod \"d59ef92f-e353-45e2-9512-70791c54d288\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " Nov 02 22:32:07.536535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536488 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59ef92f-e353-45e2-9512-70791c54d288-tls-certs\" (UniqueName: \"kubernetes.io/secret/d59ef92f-e353-45e2-9512-70791c54d288-tls-certs\") pod \"d59ef92f-e353-45e2-9512-70791c54d288\" (UID: \"d59ef92f-e353-45e2-9512-70791c54d288\") " Nov 02 22:32:07.536535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536523 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-tmp-dir" pod "d59ef92f-e353-45e2-9512-70791c54d288" (UID: "d59ef92f-e353-45e2-9512-70791c54d288"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.536853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536774 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-home" pod "d59ef92f-e353-45e2-9512-70791c54d288" (UID: "d59ef92f-e353-45e2-9512-70791c54d288"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.536853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.536788 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-model-cache" pod "d59ef92f-e353-45e2-9512-70791c54d288" (UID: "d59ef92f-e353-45e2-9512-70791c54d288"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.537378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.537297 5109 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-tmp-dir\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.537378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.537319 5109 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-model-cache\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.537378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.537332 5109 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-home\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.539806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.539765 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-dshm" pod "d59ef92f-e353-45e2-9512-70791c54d288" (UID: "d59ef92f-e353-45e2-9512-70791c54d288"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.539962 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.539937 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59ef92f-e353-45e2-9512-70791c54d288-tls-certs" pod "d59ef92f-e353-45e2-9512-70791c54d288" (UID: "d59ef92f-e353-45e2-9512-70791c54d288"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:07.540472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.540449 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59ef92f-e353-45e2-9512-70791c54d288-kube-api-access-v2lh9" pod "d59ef92f-e353-45e2-9512-70791c54d288" (UID: "d59ef92f-e353-45e2-9512-70791c54d288"). InnerVolumeSpecName "kube-api-access-v2lh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:07.564230 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.564192 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn"] Nov 02 22:32:07.570738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.570705 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/premium-w4-3224ea-kserve-84d67555b9-fgmhn"] Nov 02 22:32:07.652832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.641747 5109 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59ef92f-e353-45e2-9512-70791c54d288-dshm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.652832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.641781 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2lh9\" (UniqueName: \"kubernetes.io/projected/d59ef92f-e353-45e2-9512-70791c54d288-kube-api-access-v2lh9\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.652832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.641828 5109 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59ef92f-e353-45e2-9512-70791c54d288-tls-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.676059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.676021 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x"] Nov 02 22:32:07.680837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.680810 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-w4-3224ea-kserve-5dcc76b69b-2h92x"] Nov 02 22:32:07.791858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.791826 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp"] Nov 02 22:32:07.796359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.796335 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3224ea/distinct-2-w4-3224ea-kserve-749768b767-99llp"] Nov 02 22:32:07.894005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.893975 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" path="/var/lib/kubelet/pods/47c2f9cc-3ba0-4891-9790-d86df9dac639/volumes" Nov 02 22:32:07.894483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.894469 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3d9868-5f5a-413a-a60f-fad22d6cb6c5" path="/var/lib/kubelet/pods/7c3d9868-5f5a-413a-a60f-fad22d6cb6c5/volumes" Nov 02 22:32:07.894937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.894923 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" path="/var/lib/kubelet/pods/8c658010-2eae-42cf-8a70-f08f3014976c/volumes" Nov 02 22:32:07.895695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:07.895675 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" path="/var/lib/kubelet/pods/9f43ab2f-5a4c-4759-9212-ae3ab7f7d463/volumes" Nov 02 22:32:08.344211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:08.344184 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" Nov 02 22:32:08.344444 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:08.344176 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv" event={"ID":"d59ef92f-e353-45e2-9512-70791c54d288","Type":"ContainerDied","Data":"05ad927fe022e568c862236ba0aad2de50994f4d3178b4450f399bab6f5fd720"} Nov 02 22:32:08.344444 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:08.344333 5109 scope.go:122] "RemoveContainer" containerID="58fa74e1945c4dac3161ab3d3f8e38a8e0457459c8cbe46e7400fe9922911af6" Nov 02 22:32:08.513661 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:08.513630 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv"] Nov 02 22:32:08.518034 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:08.518012 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-w1-ca85db-kserve-659f84f889-qkfcv"] Nov 02 22:32:09.054450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:09.054409 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn"] Nov 02 22:32:09.058656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:09.058631 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e58370-vdcwn"] Nov 02 22:32:09.894187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:09.894151 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59ef92f-e353-45e2-9512-70791c54d288" path="/var/lib/kubelet/pods/d59ef92f-e353-45e2-9512-70791c54d288/volumes" Nov 02 22:32:09.894628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:09.894616 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f077906d-8b76-4065-81f4-5c2e911266a7" path="/var/lib/kubelet/pods/f077906d-8b76-4065-81f4-5c2e911266a7/volumes" Nov 02 22:32:10.053510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.053477 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx"] Nov 02 22:32:10.103156 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.103113 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66600c7d-rs5lx"] Nov 02 22:32:10.107907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.107879 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82"] Nov 02 22:32:10.108691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108670 5109 cpu_manager.go:414] "containerMap: removing container" podUID="92f3878b-4a2a-4689-9a7b-0455af40d585" containerName="revoke-keys" Nov 02 22:32:10.108691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108691 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f3878b-4a2a-4689-9a7b-0455af40d585" containerName="revoke-keys" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108707 5109 cpu_manager.go:414] "containerMap: removing container" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108714 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108731 5109 cpu_manager.go:414] "containerMap: removing container" podUID="54750126-b5a6-4de7-82a3-b32c71bc6f45" containerName="revoke-keys" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108738 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54750126-b5a6-4de7-82a3-b32c71bc6f45" containerName="revoke-keys" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108762 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108769 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108798 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d3a5d192-58f0-4d9f-8111-1308242009bd" containerName="payload-processing" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108804 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a5d192-58f0-4d9f-8111-1308242009bd" containerName="payload-processing" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108843 5109 cpu_manager.go:414] "containerMap: removing container" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108850 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108869 5109 cpu_manager.go:414] "containerMap: removing container" podUID="811c907d-0581-4d24-adff-5d43ebd3abdb" containerName="revoke-keys" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108876 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811c907d-0581-4d24-adff-5d43ebd3abdb" containerName="revoke-keys" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108893 5109 cpu_manager.go:414] "containerMap: removing container" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" Nov 02 22:32:10.108931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.108899 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109066 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f43ab2f-5a4c-4759-9212-ae3ab7f7d463" containerName="main" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109101 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f3878b-4a2a-4689-9a7b-0455af40d585" containerName="revoke-keys" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109113 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c2f9cc-3ba0-4891-9790-d86df9dac639" containerName="main" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109123 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="54750126-b5a6-4de7-82a3-b32c71bc6f45" containerName="revoke-keys" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109133 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59ef92f-e353-45e2-9512-70791c54d288" containerName="main" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109145 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a5d192-58f0-4d9f-8111-1308242009bd" containerName="payload-processing" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109159 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c658010-2eae-42cf-8a70-f08f3014976c" containerName="main" Nov 02 22:32:10.109655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.109170 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="811c907d-0581-4d24-adff-5d43ebd3abdb" containerName="revoke-keys" Nov 02 22:32:10.115219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.115199 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.161060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.160972 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82"] Nov 02 22:32:10.168350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.168319 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:32:10.168615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.168375 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:32:10.194989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.194953 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.195198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.195115 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xr7\" (UniqueName: \"kubernetes.io/projected/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-kube-api-access-g2xr7\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.195301 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.195235 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.296368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.296332 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xr7\" (UniqueName: \"kubernetes.io/projected/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-kube-api-access-g2xr7\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.296613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.296415 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.296695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.296640 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.297172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.297154 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.299181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.299158 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.358453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.358409 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xr7\" (UniqueName: \"kubernetes.io/projected/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-kube-api-access-g2xr7\") pod \"maas-api-revoke-keys-e2e-ait-fb670683-wrq82\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.431221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.431126 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:10.810522 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:32:10.810488 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd91f3cb_acc8_4005_871b_f9a4699bc8dd.slice/crio-bb4d49c33bdf9ffc6315267e7a6b5c7a61160195ef13729b94a193fe79fad03f WatchSource:0}: Error finding container bb4d49c33bdf9ffc6315267e7a6b5c7a61160195ef13729b94a193fe79fad03f: Status 404 returned error can't find the container with id bb4d49c33bdf9ffc6315267e7a6b5c7a61160195ef13729b94a193fe79fad03f Nov 02 22:32:10.814375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:10.814349 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82"] Nov 02 22:32:11.379113 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:11.379076 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" event={"ID":"cd91f3cb-acc8-4005-871b-f9a4699bc8dd","Type":"ContainerStarted","Data":"145ec5e76c99cd5164b92740bf56753822976048597784a088cdbce710e4e88f"} Nov 02 22:32:11.379623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:11.379144 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" event={"ID":"cd91f3cb-acc8-4005-871b-f9a4699bc8dd","Type":"ContainerStarted","Data":"bb4d49c33bdf9ffc6315267e7a6b5c7a61160195ef13729b94a193fe79fad03f"} Nov 02 22:32:11.500653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:11.500498 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" podStartSLOduration=1.500481969 podStartE2EDuration="1.500481969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:11.495527251 +0000 UTC m=+1567.760520470" watchObservedRunningTime="2026-11-02 22:32:11.500481969 +0000 UTC m=+1567.765475189" Nov 02 22:32:11.894056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:11.894020 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a513f334-622c-4f92-bad0-09be1fd7f6db" path="/var/lib/kubelet/pods/a513f334-622c-4f92-bad0-09be1fd7f6db/volumes" Nov 02 22:32:11.921732 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:11.921690 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf"] Nov 02 22:32:12.023613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:12.023558 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn"] Nov 02 22:32:12.556603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:12.552932 5109 generic.go:356] "Generic (PLEG): container finished" podID="cd91f3cb-acc8-4005-871b-f9a4699bc8dd" containerID="145ec5e76c99cd5164b92740bf56753822976048597784a088cdbce710e4e88f" exitCode=0 Nov 02 22:32:12.556603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:12.553020 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" event={"ID":"cd91f3cb-acc8-4005-871b-f9a4699bc8dd","Type":"ContainerDied","Data":"145ec5e76c99cd5164b92740bf56753822976048597784a088cdbce710e4e88f"} Nov 02 22:32:14.281442 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.281195 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7"] Nov 02 22:32:14.282259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.282225 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" containerID="cri-o://ab71ee402ba7258a547b5d439db42eeb58654efd01d6c647125b2d47c4c631f2" gracePeriod=30 Nov 02 22:32:14.322371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.322333 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg"] Nov 02 22:32:14.330005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.329939 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" containerID="cri-o://5fd5d57c28e636dc0e04f62d3686a04a159996b121a6f0dc81aaa3f0d9ec3d2a" gracePeriod=30 Nov 02 22:32:14.337668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.337636 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng"] Nov 02 22:32:14.338191 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.338160 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" containerID="cri-o://b846111df9472d34d34163dcd18be102ba5d46f53e8fea7b43d4585a1d26504f" gracePeriod=30 Nov 02 22:32:14.444152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.444116 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:14.588127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.588035 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-openshift-ca\") pod \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " Nov 02 22:32:14.588127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.588114 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-xks-ca\") pod \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " Nov 02 22:32:14.588324 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.588268 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-kube-api-access-g2xr7\" (UniqueName: \"kubernetes.io/projected/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-kube-api-access-g2xr7\") pod \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\" (UID: \"cd91f3cb-acc8-4005-871b-f9a4699bc8dd\") " Nov 02 22:32:14.588481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.588455 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-openshift-ca" pod "cd91f3cb-acc8-4005-871b-f9a4699bc8dd" (UID: "cd91f3cb-acc8-4005-871b-f9a4699bc8dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:14.588853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.588833 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:14.590496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.590469 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-xks-ca" pod "cd91f3cb-acc8-4005-871b-f9a4699bc8dd" (UID: "cd91f3cb-acc8-4005-871b-f9a4699bc8dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:14.590556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.590526 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-kube-api-access-g2xr7" pod "cd91f3cb-acc8-4005-871b-f9a4699bc8dd" (UID: "cd91f3cb-acc8-4005-871b-f9a4699bc8dd"). InnerVolumeSpecName "kube-api-access-g2xr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:14.633735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.633700 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" Nov 02 22:32:14.633735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.633726 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82" event={"ID":"cd91f3cb-acc8-4005-871b-f9a4699bc8dd","Type":"ContainerDied","Data":"bb4d49c33bdf9ffc6315267e7a6b5c7a61160195ef13729b94a193fe79fad03f"} Nov 02 22:32:14.633949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.633770 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb4d49c33bdf9ffc6315267e7a6b5c7a61160195ef13729b94a193fe79fad03f" Nov 02 22:32:14.690744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.690708 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:14.690901 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:14.690758 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xr7\" (UniqueName: \"kubernetes.io/projected/cd91f3cb-acc8-4005-871b-f9a4699bc8dd-kube-api-access-g2xr7\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:15.908721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:15.908681 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" podUID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:32:16.027735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.027697 5109 scope.go:122] "RemoveContainer" containerID="874f4bf2ff1060912eaec8cf373a03cd14aab009b2a5efa9a27b3defa5caac41" Nov 02 22:32:16.040801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.040778 5109 scope.go:122] "RemoveContainer" containerID="cd18747f4ebb39eb2eb2e7425d2ecc2f77349aff1a5fcf436e05c97ebab9337f" Nov 02 22:32:16.051949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.051923 5109 scope.go:122] "RemoveContainer" containerID="263aae860fda7a73907baca01fe72bc0195c67167a9f3f52f8bc28d481d3292d" Nov 02 22:32:16.063759 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.063739 5109 scope.go:122] "RemoveContainer" containerID="562345fd491759a47236d0c0c0d4d0e583e0e276252f8c69ba1211c0a1d9a395" Nov 02 22:32:16.074912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.074884 5109 scope.go:122] "RemoveContainer" containerID="b4a87ed917f8bd0af2d15e5a70bab389172535560eae1416e496bbb3a3bf3678" Nov 02 22:32:16.085433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.085408 5109 scope.go:122] "RemoveContainer" containerID="13a624a620b51479f6e162399ad0d423b4a0c6adc7ed95ac1df7eb55931a1cc6" Nov 02 22:32:16.095698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.095675 5109 scope.go:122] "RemoveContainer" containerID="001b506a1077449e2bb21b9c45c4b9eb2fc22a44f060c49a9d0f6e60f3482643" Nov 02 22:32:16.922387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:16.922344 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" podUID="4f88dc55-b356-4640-b632-8c64cb9e3a42" containerName="payload-processing" containerID="cri-o://081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec" gracePeriod=25 Nov 02 22:32:17.024581 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.024496 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" podUID="66f9affa-b826-4bfb-a70c-4401a3a4fa26" containerName="payload-pre-processing" containerID="cri-o://04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626" gracePeriod=25 Nov 02 22:32:17.231314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.231283 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:32:17.301373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.301344 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:32:17.432363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.432333 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f9affa-b826-4bfb-a70c-4401a3a4fa26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f9affa-b826-4bfb-a70c-4401a3a4fa26-plugins-config-volume\") pod \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " Nov 02 22:32:17.432609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.432458 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f88dc55-b356-4640-b632-8c64cb9e3a42-kube-api-access-mc7cc\" (UniqueName: \"kubernetes.io/projected/4f88dc55-b356-4640-b632-8c64cb9e3a42-kube-api-access-mc7cc\") pod \"4f88dc55-b356-4640-b632-8c64cb9e3a42\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " Nov 02 22:32:17.432609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.432534 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f88dc55-b356-4640-b632-8c64cb9e3a42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f88dc55-b356-4640-b632-8c64cb9e3a42-plugins-config-volume\") pod \"4f88dc55-b356-4640-b632-8c64cb9e3a42\" (UID: \"4f88dc55-b356-4640-b632-8c64cb9e3a42\") " Nov 02 22:32:17.432748 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.432731 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f9affa-b826-4bfb-a70c-4401a3a4fa26-kube-api-access-vwbd4\" (UniqueName: \"kubernetes.io/projected/66f9affa-b826-4bfb-a70c-4401a3a4fa26-kube-api-access-vwbd4\") pod \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\" (UID: \"66f9affa-b826-4bfb-a70c-4401a3a4fa26\") " Nov 02 22:32:17.432881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.432859 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f9affa-b826-4bfb-a70c-4401a3a4fa26-plugins-config-volume" pod "66f9affa-b826-4bfb-a70c-4401a3a4fa26" (UID: "66f9affa-b826-4bfb-a70c-4401a3a4fa26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:17.432965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.432910 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f88dc55-b356-4640-b632-8c64cb9e3a42-plugins-config-volume" pod "4f88dc55-b356-4640-b632-8c64cb9e3a42" (UID: "4f88dc55-b356-4640-b632-8c64cb9e3a42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:17.433281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.433261 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f9affa-b826-4bfb-a70c-4401a3a4fa26-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:17.433281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.433281 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f88dc55-b356-4640-b632-8c64cb9e3a42-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:17.434989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.434964 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f88dc55-b356-4640-b632-8c64cb9e3a42-kube-api-access-mc7cc" pod "4f88dc55-b356-4640-b632-8c64cb9e3a42" (UID: "4f88dc55-b356-4640-b632-8c64cb9e3a42"). InnerVolumeSpecName "kube-api-access-mc7cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:17.435069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.435004 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f9affa-b826-4bfb-a70c-4401a3a4fa26-kube-api-access-vwbd4" pod "66f9affa-b826-4bfb-a70c-4401a3a4fa26" (UID: "66f9affa-b826-4bfb-a70c-4401a3a4fa26"). InnerVolumeSpecName "kube-api-access-vwbd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:17.534468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.534424 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc7cc\" (UniqueName: \"kubernetes.io/projected/4f88dc55-b356-4640-b632-8c64cb9e3a42-kube-api-access-mc7cc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:17.534468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.534464 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwbd4\" (UniqueName: \"kubernetes.io/projected/66f9affa-b826-4bfb-a70c-4401a3a4fa26-kube-api-access-vwbd4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:17.661429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.661391 5109 generic.go:356] "Generic (PLEG): container finished" podID="66f9affa-b826-4bfb-a70c-4401a3a4fa26" containerID="04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626" exitCode=0 Nov 02 22:32:17.661643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.661435 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" Nov 02 22:32:17.661643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.661477 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" event={"ID":"66f9affa-b826-4bfb-a70c-4401a3a4fa26","Type":"ContainerDied","Data":"04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626"} Nov 02 22:32:17.661643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.661532 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn" event={"ID":"66f9affa-b826-4bfb-a70c-4401a3a4fa26","Type":"ContainerDied","Data":"abe7f2d85e766f1af2bcacb7df0883b6bf03466393b14cd6f34eb696dce0db97"} Nov 02 22:32:17.661643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.661555 5109 scope.go:122] "RemoveContainer" containerID="04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626" Nov 02 22:32:17.663336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.663271 5109 generic.go:356] "Generic (PLEG): container finished" podID="4f88dc55-b356-4640-b632-8c64cb9e3a42" containerID="081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec" exitCode=0 Nov 02 22:32:17.663336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.663296 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" event={"ID":"4f88dc55-b356-4640-b632-8c64cb9e3a42","Type":"ContainerDied","Data":"081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec"} Nov 02 22:32:17.663457 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.663345 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" event={"ID":"4f88dc55-b356-4640-b632-8c64cb9e3a42","Type":"ContainerDied","Data":"067e68bd4003cbdbc3e2e8df38df19b237a708f9dc60d0540f200078cbed1a98"} Nov 02 22:32:17.663507 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.663490 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf" Nov 02 22:32:17.678024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.677617 5109 scope.go:122] "RemoveContainer" containerID="04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626" Nov 02 22:32:17.680411 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:17.680380 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626\": container with ID starting with 04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626 not found: ID does not exist" containerID="04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626" Nov 02 22:32:17.680524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.680420 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626"} err="failed to get container status \"04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626\": rpc error: code = NotFound desc = could not find container \"04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626\": container with ID starting with 04520bd3ef76865e4ec4b7fcd3bda543999821d35677f88d4284c6acd43a9626 not found: ID does not exist" Nov 02 22:32:17.680524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.680444 5109 scope.go:122] "RemoveContainer" containerID="081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec" Nov 02 22:32:17.691374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.691355 5109 scope.go:122] "RemoveContainer" containerID="081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec" Nov 02 22:32:17.691703 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:17.691680 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec\": container with ID starting with 081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec not found: ID does not exist" containerID="081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec" Nov 02 22:32:17.691783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.691715 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec"} err="failed to get container status \"081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec\": rpc error: code = NotFound desc = could not find container \"081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec\": container with ID starting with 081c57d855babd31e554c7f85132e8cf891033c234469eb98c04e709edb3b4ec not found: ID does not exist" Nov 02 22:32:17.831758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.831673 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn"] Nov 02 22:32:17.835958 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.835935 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2663d62e-77974d57f9-qdtdn"] Nov 02 22:32:17.893963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.893929 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f9affa-b826-4bfb-a70c-4401a3a4fa26" path="/var/lib/kubelet/pods/66f9affa-b826-4bfb-a70c-4401a3a4fa26/volumes" Nov 02 22:32:17.947075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.947047 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf"] Nov 02 22:32:17.953269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:17.953239 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2663d62e-6d594c77cc-fmtdf"] Nov 02 22:32:19.698106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.698073 5109 generic.go:356] "Generic (PLEG): container finished" podID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerID="b846111df9472d34d34163dcd18be102ba5d46f53e8fea7b43d4585a1d26504f" exitCode=0 Nov 02 22:32:19.698621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.698460 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" event={"ID":"bb24859c-e64b-4f72-b230-c85c1b09c596","Type":"ContainerDied","Data":"b846111df9472d34d34163dcd18be102ba5d46f53e8fea7b43d4585a1d26504f"} Nov 02 22:32:19.700628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.700558 5109 generic.go:356] "Generic (PLEG): container finished" podID="9df8261b-156f-4a46-9155-711018c994b1" containerID="5fd5d57c28e636dc0e04f62d3686a04a159996b121a6f0dc81aaa3f0d9ec3d2a" exitCode=0 Nov 02 22:32:19.700794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.700680 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" event={"ID":"9df8261b-156f-4a46-9155-711018c994b1","Type":"ContainerDied","Data":"5fd5d57c28e636dc0e04f62d3686a04a159996b121a6f0dc81aaa3f0d9ec3d2a"} Nov 02 22:32:19.700794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.700717 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" event={"ID":"9df8261b-156f-4a46-9155-711018c994b1","Type":"ContainerDied","Data":"3accec89dc07e7788bf87f9b89b42bc3189c353e6fcb63036198e846e96c8e95"} Nov 02 22:32:19.700794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.700737 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3accec89dc07e7788bf87f9b89b42bc3189c353e6fcb63036198e846e96c8e95" Nov 02 22:32:19.702856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.702829 5109 generic.go:356] "Generic (PLEG): container finished" podID="62782404-05cd-4b9c-837c-038514b8c096" containerID="ab71ee402ba7258a547b5d439db42eeb58654efd01d6c647125b2d47c4c631f2" exitCode=0 Nov 02 22:32:19.702974 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.702909 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" event={"ID":"62782404-05cd-4b9c-837c-038514b8c096","Type":"ContainerDied","Data":"ab71ee402ba7258a547b5d439db42eeb58654efd01d6c647125b2d47c4c631f2"} Nov 02 22:32:19.703137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.703116 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:32:19.733813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.733778 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:32:19.774350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.774319 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:32:19.776459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776432 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-istio-token\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-istio-token\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.776627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776488 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-envoy\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.776627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776533 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-credential-socket\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.776627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776584 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df8261b-156f-4a46-9155-711018c994b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df8261b-156f-4a46-9155-711018c994b1-istio-podinfo\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.776798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776630 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62782404-05cd-4b9c-837c-038514b8c096-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62782404-05cd-4b9c-837c-038514b8c096-istio-podinfo\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.776798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776708 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-credential-socket\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.776798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776730 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-socket\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.776798 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776764 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-certs\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.777011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776841 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-envoy\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.777011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776839 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-credential-socket" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.777011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776885 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-istio-token\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-istio-token\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.777011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776915 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-data\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.777011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.776986 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-certs\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.777276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777048 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df8261b-156f-4a46-9155-711018c994b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df8261b-156f-4a46-9155-711018c994b1-istiod-ca-cert\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.777276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777068 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-socket" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.777276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777136 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-data\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.777276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777186 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62782404-05cd-4b9c-837c-038514b8c096-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62782404-05cd-4b9c-837c-038514b8c096-istiod-ca-cert\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.777276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777259 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-credential-socket" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.777719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777286 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-certs" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.777719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777475 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-certs" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.777719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777536 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-kube-api-access-q75fm\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-kube-api-access-q75fm\") pod \"9df8261b-156f-4a46-9155-711018c994b1\" (UID: \"9df8261b-156f-4a46-9155-711018c994b1\") " Nov 02 22:32:19.777719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777619 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62782404-05cd-4b9c-837c-038514b8c096-istiod-ca-cert" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:19.777719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777677 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-kube-api-access-9jlzn\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-kube-api-access-9jlzn\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.777995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.777731 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-socket\") pod \"62782404-05cd-4b9c-837c-038514b8c096\" (UID: \"62782404-05cd-4b9c-837c-038514b8c096\") " Nov 02 22:32:19.778144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778123 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-data" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.778144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778131 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-data" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.778365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778331 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-socket" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.778498 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778365 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df8261b-156f-4a46-9155-711018c994b1-istiod-ca-cert" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:19.778850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778826 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778855 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778871 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778888 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778901 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778919 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778933 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778947 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df8261b-156f-4a46-9155-711018c994b1-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.778966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778961 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.779325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.778975 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62782404-05cd-4b9c-837c-038514b8c096-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.780140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.780107 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-istio-token" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:19.780227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.780167 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df8261b-156f-4a46-9155-711018c994b1-istio-podinfo" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:19.780227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.780164 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-envoy" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.780536 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.780518 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62782404-05cd-4b9c-837c-038514b8c096-istio-podinfo" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:19.780629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.780558 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-envoy" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.780809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.780784 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-istio-token" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:19.781106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.781088 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-kube-api-access-9jlzn" pod "62782404-05cd-4b9c-837c-038514b8c096" (UID: "62782404-05cd-4b9c-837c-038514b8c096"). InnerVolumeSpecName "kube-api-access-9jlzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:19.781940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.781922 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-kube-api-access-q75fm" pod "9df8261b-156f-4a46-9155-711018c994b1" (UID: "9df8261b-156f-4a46-9155-711018c994b1"). InnerVolumeSpecName "kube-api-access-q75fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:19.880050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.879962 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-kube-api-access-wtw5m\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-kube-api-access-wtw5m\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880004 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-certs\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880083 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-token\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-token\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880134 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb24859c-e64b-4f72-b230-c85c1b09c596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb24859c-e64b-4f72-b230-c85c1b09c596-istiod-ca-cert\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880171 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-podinfo\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880297 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-credential-socket\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880374 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-socket\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880421 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-certs" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.880653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880437 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-envoy\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.880653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880542 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-data\") pod \"bb24859c-e64b-4f72-b230-c85c1b09c596\" (UID: \"bb24859c-e64b-4f72-b230-c85c1b09c596\") " Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880658 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-credential-socket" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880820 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-socket" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.880997 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb24859c-e64b-4f72-b230-c85c1b09c596-istiod-ca-cert" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881064 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-data" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881899 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881924 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62782404-05cd-4b9c-837c-038514b8c096-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881940 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.881949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881954 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881967 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881981 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75fm\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-kube-api-access-q75fm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.881996 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jlzn\" (UniqueName: \"kubernetes.io/projected/62782404-05cd-4b9c-837c-038514b8c096-kube-api-access-9jlzn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882011 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882024 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df8261b-156f-4a46-9155-711018c994b1-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882037 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df8261b-156f-4a46-9155-711018c994b1-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882052 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df8261b-156f-4a46-9155-711018c994b1-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882064 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb24859c-e64b-4f72-b230-c85c1b09c596-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882078 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62782404-05cd-4b9c-837c-038514b8c096-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.882988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882895 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-token" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:19.882988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.882940 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-kube-api-access-wtw5m" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "kube-api-access-wtw5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:19.883356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.883339 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-podinfo" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:19.884870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.884846 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-envoy" pod "bb24859c-e64b-4f72-b230-c85c1b09c596" (UID: "bb24859c-e64b-4f72-b230-c85c1b09c596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:19.896306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.896271 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f88dc55-b356-4640-b632-8c64cb9e3a42" path="/var/lib/kubelet/pods/4f88dc55-b356-4640-b632-8c64cb9e3a42/volumes" Nov 02 22:32:19.983082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.983046 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.983082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.983075 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtw5m\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-kube-api-access-wtw5m\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.983082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.983086 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:19.983333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:19.983095 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb24859c-e64b-4f72-b230-c85c1b09c596-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:20.713619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.713495 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" event={"ID":"bb24859c-e64b-4f72-b230-c85c1b09c596","Type":"ContainerDied","Data":"8d3f56d56282cda7752ea63a0e6390271d9e7de2c3820c9be5fb21d6e808d29c"} Nov 02 22:32:20.713619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.713527 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng" Nov 02 22:32:20.713619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.713618 5109 scope.go:122] "RemoveContainer" containerID="b846111df9472d34d34163dcd18be102ba5d46f53e8fea7b43d4585a1d26504f" Nov 02 22:32:20.715751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.715718 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" event={"ID":"62782404-05cd-4b9c-837c-038514b8c096","Type":"ContainerDied","Data":"7cc1cddb5312a970fac94424c04f8b12c890b174e64b3d69d566cae63e53a357"} Nov 02 22:32:20.715751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.715741 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg" Nov 02 22:32:20.715923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.715735 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7" Nov 02 22:32:20.726712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.726499 5109 scope.go:122] "RemoveContainer" containerID="ab71ee402ba7258a547b5d439db42eeb58654efd01d6c647125b2d47c4c631f2" Nov 02 22:32:20.882918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.882881 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng"] Nov 02 22:32:20.932837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.932800 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-bmfng"] Nov 02 22:32:20.998769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:20.998671 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7"] Nov 02 22:32:21.003614 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:21.003556 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-8g5k7"] Nov 02 22:32:21.135186 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:21.135148 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg"] Nov 02 22:32:21.190755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:21.190718 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-vwnrg"] Nov 02 22:32:21.894407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:21.894366 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62782404-05cd-4b9c-837c-038514b8c096" path="/var/lib/kubelet/pods/62782404-05cd-4b9c-837c-038514b8c096/volumes" Nov 02 22:32:21.894978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:21.894961 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df8261b-156f-4a46-9155-711018c994b1" path="/var/lib/kubelet/pods/9df8261b-156f-4a46-9155-711018c994b1/volumes" Nov 02 22:32:21.895397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:21.895383 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" path="/var/lib/kubelet/pods/bb24859c-e64b-4f72-b230-c85c1b09c596/volumes" Nov 02 22:32:22.300832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:22.300794 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h"] Nov 02 22:32:22.422559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:22.417858 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql"] Nov 02 22:32:25.907633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:25.907558 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" podUID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:32:26.492604 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:26.492543 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:32:27.145648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.145614 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:32:27.166864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.166835 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-kube-api-access-qn8tl\" (UniqueName: \"kubernetes.io/projected/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-kube-api-access-qn8tl\") pod \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " Nov 02 22:32:27.167059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.167012 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-plugins-config-volume\") pod \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\" (UID: \"f9cad0f9-cd2a-42e7-b4fb-5260667ec171\") " Nov 02 22:32:27.167379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.167352 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-plugins-config-volume" pod "f9cad0f9-cd2a-42e7-b4fb-5260667ec171" (UID: "f9cad0f9-cd2a-42e7-b4fb-5260667ec171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:27.167735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.167716 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:27.169171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.169148 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-kube-api-access-qn8tl" pod "f9cad0f9-cd2a-42e7-b4fb-5260667ec171" (UID: "f9cad0f9-cd2a-42e7-b4fb-5260667ec171"). InnerVolumeSpecName "kube-api-access-qn8tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:27.269617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.269583 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8tl\" (UniqueName: \"kubernetes.io/projected/f9cad0f9-cd2a-42e7-b4fb-5260667ec171-kube-api-access-qn8tl\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:27.301626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.301544 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" podUID="2addc7c0-75ea-468a-964d-b47cb6ed7303" containerName="payload-processing" containerID="cri-o://c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76" gracePeriod=25 Nov 02 22:32:27.418330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.418289 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" podUID="0b93c72f-8c9b-4416-8d21-b8d2f894b19a" containerName="payload-pre-processing" containerID="cri-o://fe570f8b5d9f25e5cf067ae228dfcc733493be43b4c2dc5bf9150bffce9cab5c" gracePeriod=25 Nov 02 22:32:27.760314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.759884 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:32:27.850372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.848420 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2addc7c0-75ea-468a-964d-b47cb6ed7303-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2addc7c0-75ea-468a-964d-b47cb6ed7303-plugins-config-volume\") pod \"2addc7c0-75ea-468a-964d-b47cb6ed7303\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " Nov 02 22:32:27.850372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.848860 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2addc7c0-75ea-468a-964d-b47cb6ed7303-kube-api-access-22hkh\" (UniqueName: \"kubernetes.io/projected/2addc7c0-75ea-468a-964d-b47cb6ed7303-kube-api-access-22hkh\") pod \"2addc7c0-75ea-468a-964d-b47cb6ed7303\" (UID: \"2addc7c0-75ea-468a-964d-b47cb6ed7303\") " Nov 02 22:32:27.850990 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.850936 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2addc7c0-75ea-468a-964d-b47cb6ed7303-plugins-config-volume" pod "2addc7c0-75ea-468a-964d-b47cb6ed7303" (UID: "2addc7c0-75ea-468a-964d-b47cb6ed7303"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:27.896870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.896784 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2addc7c0-75ea-468a-964d-b47cb6ed7303-kube-api-access-22hkh" pod "2addc7c0-75ea-468a-964d-b47cb6ed7303" (UID: "2addc7c0-75ea-468a-964d-b47cb6ed7303"). InnerVolumeSpecName "kube-api-access-22hkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:27.907933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.907877 5109 generic.go:356] "Generic (PLEG): container finished" podID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" containerID="8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71" exitCode=0 Nov 02 22:32:27.908277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.908104 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" Nov 02 22:32:27.916628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.913848 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" event={"ID":"f9cad0f9-cd2a-42e7-b4fb-5260667ec171","Type":"ContainerDied","Data":"8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71"} Nov 02 22:32:27.916628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.913913 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn" event={"ID":"f9cad0f9-cd2a-42e7-b4fb-5260667ec171","Type":"ContainerDied","Data":"52078d3e560c3356e9e2340f8dd54060e9e37ef04d6bbb4d49639879e9a3f8f7"} Nov 02 22:32:27.916628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.913942 5109 scope.go:122] "RemoveContainer" containerID="8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71" Nov 02 22:32:27.917408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.917357 5109 generic.go:356] "Generic (PLEG): container finished" podID="0b93c72f-8c9b-4416-8d21-b8d2f894b19a" containerID="fe570f8b5d9f25e5cf067ae228dfcc733493be43b4c2dc5bf9150bffce9cab5c" exitCode=0 Nov 02 22:32:27.917544 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.917484 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" event={"ID":"0b93c72f-8c9b-4416-8d21-b8d2f894b19a","Type":"ContainerDied","Data":"fe570f8b5d9f25e5cf067ae228dfcc733493be43b4c2dc5bf9150bffce9cab5c"} Nov 02 22:32:27.925860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.921007 5109 generic.go:356] "Generic (PLEG): container finished" podID="2addc7c0-75ea-468a-964d-b47cb6ed7303" containerID="c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76" exitCode=0 Nov 02 22:32:27.925860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.921158 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" event={"ID":"2addc7c0-75ea-468a-964d-b47cb6ed7303","Type":"ContainerDied","Data":"c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76"} Nov 02 22:32:27.925860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.921206 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" event={"ID":"2addc7c0-75ea-468a-964d-b47cb6ed7303","Type":"ContainerDied","Data":"6cd178cd6e38c03ec5adecc35e49afda61dfced2ab2fec94518c840f6ff649e8"} Nov 02 22:32:27.925860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.921285 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h" Nov 02 22:32:27.946831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.946799 5109 scope.go:122] "RemoveContainer" containerID="8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71" Nov 02 22:32:27.947235 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:27.947207 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71\": container with ID starting with 8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71 not found: ID does not exist" containerID="8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71" Nov 02 22:32:27.947332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.947248 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71"} err="failed to get container status \"8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71\": rpc error: code = NotFound desc = could not find container \"8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71\": container with ID starting with 8c2e70a7d311b47329aef25c973959dc2a6d7e4be37bff6458b33919a9abce71 not found: ID does not exist" Nov 02 22:32:27.947332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.947277 5109 scope.go:122] "RemoveContainer" containerID="c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76" Nov 02 22:32:27.950827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.950800 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22hkh\" (UniqueName: \"kubernetes.io/projected/2addc7c0-75ea-468a-964d-b47cb6ed7303-kube-api-access-22hkh\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:27.950827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.950826 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2addc7c0-75ea-468a-964d-b47cb6ed7303-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:27.960838 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.960812 5109 scope.go:122] "RemoveContainer" containerID="c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76" Nov 02 22:32:27.961277 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:27.961251 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76\": container with ID starting with c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76 not found: ID does not exist" containerID="c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76" Nov 02 22:32:27.961652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:27.961283 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76"} err="failed to get container status \"c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76\": rpc error: code = NotFound desc = could not find container \"c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76\": container with ID starting with c50a80b280fb0804d3338d4974f890e6ec5805b42ae7eea3a94634c0ca17fd76 not found: ID does not exist" Nov 02 22:32:28.090840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.090809 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn"] Nov 02 22:32:28.095594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.095536 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3224ea-6d998cb495-k55tn"] Nov 02 22:32:28.240595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.233084 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h"] Nov 02 22:32:28.255195 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.254827 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:32:28.282104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.282072 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fb670683-56bcb55d99-dfh8h"] Nov 02 22:32:28.364224 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.364120 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-plugins-config-volume\") pod \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " Nov 02 22:32:28.364400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.364370 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-kube-api-access-6s9vm\" (UniqueName: \"kubernetes.io/projected/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-kube-api-access-6s9vm\") pod \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\" (UID: \"0b93c72f-8c9b-4416-8d21-b8d2f894b19a\") " Nov 02 22:32:28.364604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.364532 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-plugins-config-volume" pod "0b93c72f-8c9b-4416-8d21-b8d2f894b19a" (UID: "0b93c72f-8c9b-4416-8d21-b8d2f894b19a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:28.365055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.365033 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:28.367146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.367115 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-kube-api-access-6s9vm" pod "0b93c72f-8c9b-4416-8d21-b8d2f894b19a" (UID: "0b93c72f-8c9b-4416-8d21-b8d2f894b19a"). InnerVolumeSpecName "kube-api-access-6s9vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:28.466751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.466714 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s9vm\" (UniqueName: \"kubernetes.io/projected/0b93c72f-8c9b-4416-8d21-b8d2f894b19a-kube-api-access-6s9vm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:28.932859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.932825 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" Nov 02 22:32:28.933086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.932815 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql" event={"ID":"0b93c72f-8c9b-4416-8d21-b8d2f894b19a","Type":"ContainerDied","Data":"169cfa62c300e3983fb770ba79571a37be6e1d03564320ca266157a73aeb71f7"} Nov 02 22:32:28.933086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:28.932987 5109 scope.go:122] "RemoveContainer" containerID="fe570f8b5d9f25e5cf067ae228dfcc733493be43b4c2dc5bf9150bffce9cab5c" Nov 02 22:32:29.100913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:29.100875 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql"] Nov 02 22:32:29.105297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:29.105270 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fb670683-865bf858b7-vrhql"] Nov 02 22:32:29.893336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:29.893299 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b93c72f-8c9b-4416-8d21-b8d2f894b19a" path="/var/lib/kubelet/pods/0b93c72f-8c9b-4416-8d21-b8d2f894b19a/volumes" Nov 02 22:32:29.893830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:29.893815 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2addc7c0-75ea-468a-964d-b47cb6ed7303" path="/var/lib/kubelet/pods/2addc7c0-75ea-468a-964d-b47cb6ed7303/volumes" Nov 02 22:32:29.902120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:29.894802 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" path="/var/lib/kubelet/pods/f9cad0f9-cd2a-42e7-b4fb-5260667ec171/volumes" Nov 02 22:32:36.172928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.172884 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2"] Nov 02 22:32:36.173897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173876 5109 cpu_manager.go:414] "containerMap: removing container" podUID="4f88dc55-b356-4640-b632-8c64cb9e3a42" containerName="payload-processing" Nov 02 22:32:36.173976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173899 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f88dc55-b356-4640-b632-8c64cb9e3a42" containerName="payload-processing" Nov 02 22:32:36.173976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173921 5109 cpu_manager.go:414] "containerMap: removing container" podUID="0b93c72f-8c9b-4416-8d21-b8d2f894b19a" containerName="payload-pre-processing" Nov 02 22:32:36.173976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173929 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b93c72f-8c9b-4416-8d21-b8d2f894b19a" containerName="payload-pre-processing" Nov 02 22:32:36.173976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173952 5109 cpu_manager.go:414] "containerMap: removing container" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" Nov 02 22:32:36.173976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173958 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" Nov 02 22:32:36.173976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173970 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" containerName="payload-pre-processing" Nov 02 22:32:36.173976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173976 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" containerName="payload-pre-processing" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.173998 5109 cpu_manager.go:414] "containerMap: removing container" podUID="2addc7c0-75ea-468a-964d-b47cb6ed7303" containerName="payload-processing" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174003 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2addc7c0-75ea-468a-964d-b47cb6ed7303" containerName="payload-processing" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174012 5109 cpu_manager.go:414] "containerMap: removing container" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174016 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174023 5109 cpu_manager.go:414] "containerMap: removing container" podUID="66f9affa-b826-4bfb-a70c-4401a3a4fa26" containerName="payload-pre-processing" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174027 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f9affa-b826-4bfb-a70c-4401a3a4fa26" containerName="payload-pre-processing" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174036 5109 cpu_manager.go:414] "containerMap: removing container" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174040 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174050 5109 cpu_manager.go:414] "containerMap: removing container" podUID="cd91f3cb-acc8-4005-871b-f9a4699bc8dd" containerName="revoke-keys" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174054 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd91f3cb-acc8-4005-871b-f9a4699bc8dd" containerName="revoke-keys" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174172 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd91f3cb-acc8-4005-871b-f9a4699bc8dd" containerName="revoke-keys" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174186 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f88dc55-b356-4640-b632-8c64cb9e3a42" containerName="payload-processing" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174194 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b93c72f-8c9b-4416-8d21-b8d2f894b19a" containerName="payload-pre-processing" Nov 02 22:32:36.174197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174201 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df8261b-156f-4a46-9155-711018c994b1" containerName="istio-proxy" Nov 02 22:32:36.174679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174207 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9cad0f9-cd2a-42e7-b4fb-5260667ec171" containerName="payload-pre-processing" Nov 02 22:32:36.174679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174215 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="2addc7c0-75ea-468a-964d-b47cb6ed7303" containerName="payload-processing" Nov 02 22:32:36.174679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174221 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="62782404-05cd-4b9c-837c-038514b8c096" containerName="istio-proxy" Nov 02 22:32:36.174679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174227 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f9affa-b826-4bfb-a70c-4401a3a4fa26" containerName="payload-pre-processing" Nov 02 22:32:36.174679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.174233 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb24859c-e64b-4f72-b230-c85c1b09c596" containerName="istio-proxy" Nov 02 22:32:36.179510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.179489 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.266753 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.266713 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8w7l\" (UniqueName: \"kubernetes.io/projected/bb5c1f3f-d551-449c-a605-4cda620cc9fe-kube-api-access-b8w7l\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.270948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.270900 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.275196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.275164 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.308681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.295279 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:32:36.308681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.295335 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:32:36.317993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.317950 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2"] Nov 02 22:32:36.376946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.376915 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.377121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.376962 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.377208 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.377192 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8w7l\" (UniqueName: \"kubernetes.io/projected/bb5c1f3f-d551-449c-a605-4cda620cc9fe-kube-api-access-b8w7l\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.377877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.377775 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.379538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.379516 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.438943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.438860 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8w7l\" (UniqueName: \"kubernetes.io/projected/bb5c1f3f-d551-449c-a605-4cda620cc9fe-kube-api-access-b8w7l\") pod \"maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.494558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.494518 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:36.878667 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:32:36.878635 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5c1f3f_d551_449c_a605_4cda620cc9fe.slice/crio-599f0538d46c011dcfef58f449089d35808c2ef61d2724c93813bb062b69da08 WatchSource:0}: Error finding container 599f0538d46c011dcfef58f449089d35808c2ef61d2724c93813bb062b69da08: Status 404 returned error can't find the container with id 599f0538d46c011dcfef58f449089d35808c2ef61d2724c93813bb062b69da08 Nov 02 22:32:36.882352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:36.882323 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2"] Nov 02 22:32:37.007280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:37.007250 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" event={"ID":"bb5c1f3f-d551-449c-a605-4cda620cc9fe","Type":"ContainerStarted","Data":"599f0538d46c011dcfef58f449089d35808c2ef61d2724c93813bb062b69da08"} Nov 02 22:32:38.020730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:38.020690 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" event={"ID":"bb5c1f3f-d551-449c-a605-4cda620cc9fe","Type":"ContainerDied","Data":"26bee02fababa08ab45613007818fa70afe58b449a91a59816a91e25520d4250"} Nov 02 22:32:38.021650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:38.020214 5109 generic.go:356] "Generic (PLEG): container finished" podID="bb5c1f3f-d551-449c-a605-4cda620cc9fe" containerID="26bee02fababa08ab45613007818fa70afe58b449a91a59816a91e25520d4250" exitCode=0 Nov 02 22:32:39.545143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.545111 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:39.651702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.651666 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-openshift-ca\") pod \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " Nov 02 22:32:39.651887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.651863 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-xks-ca\") pod \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " Nov 02 22:32:39.651931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.651922 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5c1f3f-d551-449c-a605-4cda620cc9fe-kube-api-access-b8w7l\" (UniqueName: \"kubernetes.io/projected/bb5c1f3f-d551-449c-a605-4cda620cc9fe-kube-api-access-b8w7l\") pod \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\" (UID: \"bb5c1f3f-d551-449c-a605-4cda620cc9fe\") " Nov 02 22:32:39.652052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.652029 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-openshift-ca" pod "bb5c1f3f-d551-449c-a605-4cda620cc9fe" (UID: "bb5c1f3f-d551-449c-a605-4cda620cc9fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:39.652375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.652358 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:39.654246 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.654214 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-xks-ca" pod "bb5c1f3f-d551-449c-a605-4cda620cc9fe" (UID: "bb5c1f3f-d551-449c-a605-4cda620cc9fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:39.654362 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.654340 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5c1f3f-d551-449c-a605-4cda620cc9fe-kube-api-access-b8w7l" pod "bb5c1f3f-d551-449c-a605-4cda620cc9fe" (UID: "bb5c1f3f-d551-449c-a605-4cda620cc9fe"). InnerVolumeSpecName "kube-api-access-b8w7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:39.753912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.753874 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5c1f3f-d551-449c-a605-4cda620cc9fe-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:39.753912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:39.753909 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8w7l\" (UniqueName: \"kubernetes.io/projected/bb5c1f3f-d551-449c-a605-4cda620cc9fe-kube-api-access-b8w7l\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:40.131266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.131228 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" Nov 02 22:32:40.131266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.131258 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2" event={"ID":"bb5c1f3f-d551-449c-a605-4cda620cc9fe","Type":"ContainerDied","Data":"599f0538d46c011dcfef58f449089d35808c2ef61d2724c93813bb062b69da08"} Nov 02 22:32:40.131668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.131312 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="599f0538d46c011dcfef58f449089d35808c2ef61d2724c93813bb062b69da08" Nov 02 22:32:40.249482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.249443 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286"] Nov 02 22:32:40.250225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.250203 5109 cpu_manager.go:414] "containerMap: removing container" podUID="bb5c1f3f-d551-449c-a605-4cda620cc9fe" containerName="revoke-keys" Nov 02 22:32:40.250225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.250225 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5c1f3f-d551-449c-a605-4cda620cc9fe" containerName="revoke-keys" Nov 02 22:32:40.250461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.250447 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5c1f3f-d551-449c-a605-4cda620cc9fe" containerName="revoke-keys" Nov 02 22:32:40.254333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.254310 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.308653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.308623 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg"] Nov 02 22:32:40.309125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.309102 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:32:40.314925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.314896 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286"] Nov 02 22:32:40.315071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.315009 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.353220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.353179 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:32:40.364462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.364432 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.364773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.364746 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvpv\" (UniqueName: \"kubernetes.io/projected/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-kube-api-access-bgvpv\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.365030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.365009 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbd6j\" (UniqueName: \"kubernetes.io/projected/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-kube-api-access-jbd6j\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.365281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.365261 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.365951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.365927 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.366133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.366034 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.372315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.372280 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg"] Nov 02 22:32:40.467506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.467420 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbd6j\" (UniqueName: \"kubernetes.io/projected/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-kube-api-access-jbd6j\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.467674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.467509 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.467674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.467626 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.467771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.467702 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.467771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.467764 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.467852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.467805 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvpv\" (UniqueName: \"kubernetes.io/projected/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-kube-api-access-bgvpv\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.468398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.468364 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.468522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.468398 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.470316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.470282 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.470422 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.470336 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.528701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.528667 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvpv\" (UniqueName: \"kubernetes.io/projected/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-kube-api-access-bgvpv\") pod \"maas-api-revoke-sub-56e240e2-49286\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.528905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.528886 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbd6j\" (UniqueName: \"kubernetes.io/projected/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-kube-api-access-jbd6j\") pod \"maas-api-revoke-sub-f1fb8ae3-fcxfg\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.575525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.568554 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:40.637596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.634852 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:40.746144 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:40.743846 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf"] Nov 02 22:32:41.262348 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:32:41.262314 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd3c167_4d7c_42c6_bf4b_9ce510fcbafe.slice/crio-ad0678e1947287a9643e114a3f663ce30bbade0ad8de211c5c69cb9a93baac65 WatchSource:0}: Error finding container ad0678e1947287a9643e114a3f663ce30bbade0ad8de211c5c69cb9a93baac65: Status 404 returned error can't find the container with id ad0678e1947287a9643e114a3f663ce30bbade0ad8de211c5c69cb9a93baac65 Nov 02 22:32:41.268630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:41.268602 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286"] Nov 02 22:32:41.311739 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:32:41.311695 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee07c24_deb2_4296_a3ad_c7bede1c7d92.slice/crio-dd7b447acc4a0b0b364c1b2ace7ed210758355391f70b728e9a3a5186ae87d87 WatchSource:0}: Error finding container dd7b447acc4a0b0b364c1b2ace7ed210758355391f70b728e9a3a5186ae87d87: Status 404 returned error can't find the container with id dd7b447acc4a0b0b364c1b2ace7ed210758355391f70b728e9a3a5186ae87d87 Nov 02 22:32:41.312950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:41.312894 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf"] Nov 02 22:32:41.317850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:41.317831 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg"] Nov 02 22:32:41.322001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:41.321981 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2663d62e-8pwwf"] Nov 02 22:32:41.389444 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:41.389404 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8"] Nov 02 22:32:41.389992 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:41.389956 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" containerID="cri-o://56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d" gracePeriod=30 Nov 02 22:32:41.906878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:41.895447 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811c907d-0581-4d24-adff-5d43ebd3abdb" path="/var/lib/kubelet/pods/811c907d-0581-4d24-adff-5d43ebd3abdb/volumes" Nov 02 22:32:42.173900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.172907 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" event={"ID":"7ee07c24-deb2-4296-a3ad-c7bede1c7d92","Type":"ContainerStarted","Data":"79b8a04a5ec1054a6620842f4a21380c21b413fb86ab1295c2e55368bcaec43f"} Nov 02 22:32:42.173900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.172984 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" event={"ID":"7ee07c24-deb2-4296-a3ad-c7bede1c7d92","Type":"ContainerStarted","Data":"dd7b447acc4a0b0b364c1b2ace7ed210758355391f70b728e9a3a5186ae87d87"} Nov 02 22:32:42.177360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.177324 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" event={"ID":"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe","Type":"ContainerStarted","Data":"af44dd9261c0996827b5e7d4a668ef071d3e8ef2994920fbe36b5f6b575e40b1"} Nov 02 22:32:42.177558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.177378 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" event={"ID":"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe","Type":"ContainerStarted","Data":"ad0678e1947287a9643e114a3f663ce30bbade0ad8de211c5c69cb9a93baac65"} Nov 02 22:32:42.213543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.209747 5109 patch_prober.go:29] interesting pod/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:32:42.213543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.209814 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:32:42.300605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.297720 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" podStartSLOduration=2.297698505 podStartE2EDuration="2.297698505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:42.295427351 +0000 UTC m=+1598.560420571" watchObservedRunningTime="2026-11-02 22:32:42.297698505 +0000 UTC m=+1598.562691725" Nov 02 22:32:42.471754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.469237 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" podStartSLOduration=2.469211239 podStartE2EDuration="2.469211239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:42.457627675 +0000 UTC m=+1598.722620894" watchObservedRunningTime="2026-11-02 22:32:42.469211239 +0000 UTC m=+1598.734204459" Nov 02 22:32:42.852779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.852742 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p"] Nov 02 22:32:42.885607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:42.885157 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3224ea-nml5p"] Nov 02 22:32:43.188939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.188836 5109 generic.go:356] "Generic (PLEG): container finished" podID="8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" containerID="af44dd9261c0996827b5e7d4a668ef071d3e8ef2994920fbe36b5f6b575e40b1" exitCode=0 Nov 02 22:32:43.189447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.188974 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" event={"ID":"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe","Type":"ContainerDied","Data":"af44dd9261c0996827b5e7d4a668ef071d3e8ef2994920fbe36b5f6b575e40b1"} Nov 02 22:32:43.191133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.191110 5109 generic.go:356] "Generic (PLEG): container finished" podID="7ee07c24-deb2-4296-a3ad-c7bede1c7d92" containerID="79b8a04a5ec1054a6620842f4a21380c21b413fb86ab1295c2e55368bcaec43f" exitCode=0 Nov 02 22:32:43.191286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.191218 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" event={"ID":"7ee07c24-deb2-4296-a3ad-c7bede1c7d92","Type":"ContainerDied","Data":"79b8a04a5ec1054a6620842f4a21380c21b413fb86ab1295c2e55368bcaec43f"} Nov 02 22:32:43.361336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.361297 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz"] Nov 02 22:32:43.366088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.366064 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.381104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.375918 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz"] Nov 02 22:32:43.524558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.524525 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.524818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.524698 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/01763f54-474e-48c8-b743-3c3e2dfdfca9-kube-api-access-ms6w5\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.524818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.524798 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.626261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.626207 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.626491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.626343 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/01763f54-474e-48c8-b743-3c3e2dfdfca9-kube-api-access-ms6w5\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.626491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.626427 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.627212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.627182 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.628993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.628972 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.687648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.687618 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/01763f54-474e-48c8-b743-3c3e2dfdfca9-kube-api-access-ms6w5\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:43.894817 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.894731 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a" path="/var/lib/kubelet/pods/042c0e84-29e2-4ac2-b8af-45a6eb6ffb0a/volumes" Nov 02 22:32:43.990594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:43.987611 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:44.437429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.437398 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:44.484620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.483097 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82"] Nov 02 22:32:44.487442 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.487411 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb670683-wrq82"] Nov 02 22:32:44.518081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.518051 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:44.532092 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.532063 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz"] Nov 02 22:32:44.548926 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:32:44.548886 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01763f54_474e_48c8_b743_3c3e2dfdfca9.slice/crio-287bc9ce3507808287c96fb71db647aa89c8176797e386cbc4e79cb43f4e688d WatchSource:0}: Error finding container 287bc9ce3507808287c96fb71db647aa89c8176797e386cbc4e79cb43f4e688d: Status 404 returned error can't find the container with id 287bc9ce3507808287c96fb71db647aa89c8176797e386cbc4e79cb43f4e688d Nov 02 22:32:44.565286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.565234 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-kube-api-access-jbd6j\" (UniqueName: \"kubernetes.io/projected/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-kube-api-access-jbd6j\") pod \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " Nov 02 22:32:44.565441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.565379 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-openshift-ca\") pod \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " Nov 02 22:32:44.565507 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.565438 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-xks-ca\") pod \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\" (UID: \"7ee07c24-deb2-4296-a3ad-c7bede1c7d92\") " Nov 02 22:32:44.565625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.565601 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-openshift-ca\") pod \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " Nov 02 22:32:44.565924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.565901 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-openshift-ca" pod "7ee07c24-deb2-4296-a3ad-c7bede1c7d92" (UID: "7ee07c24-deb2-4296-a3ad-c7bede1c7d92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:44.566443 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.566225 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:44.566773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.566737 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-openshift-ca" pod "8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" (UID: "8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:44.568870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.568841 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-xks-ca" pod "7ee07c24-deb2-4296-a3ad-c7bede1c7d92" (UID: "7ee07c24-deb2-4296-a3ad-c7bede1c7d92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:44.569405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.569380 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-kube-api-access-jbd6j" pod "7ee07c24-deb2-4296-a3ad-c7bede1c7d92" (UID: "7ee07c24-deb2-4296-a3ad-c7bede1c7d92"). InnerVolumeSpecName "kube-api-access-jbd6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:44.667338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.667250 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-kube-api-access-bgvpv\" (UniqueName: \"kubernetes.io/projected/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-kube-api-access-bgvpv\") pod \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " Nov 02 22:32:44.667514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.667341 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-xks-ca\") pod \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\" (UID: \"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe\") " Nov 02 22:32:44.668422 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.668392 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:44.668556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.668426 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:44.668556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.668444 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbd6j\" (UniqueName: \"kubernetes.io/projected/7ee07c24-deb2-4296-a3ad-c7bede1c7d92-kube-api-access-jbd6j\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:44.670180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.670151 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-xks-ca" pod "8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" (UID: "8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:44.670290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.670183 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-kube-api-access-bgvpv" pod "8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" (UID: "8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe"). InnerVolumeSpecName "kube-api-access-bgvpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:44.770158 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.770125 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgvpv\" (UniqueName: \"kubernetes.io/projected/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-kube-api-access-bgvpv\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:44.770158 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:44.770157 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:45.271724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.269478 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" event={"ID":"7ee07c24-deb2-4296-a3ad-c7bede1c7d92","Type":"ContainerDied","Data":"dd7b447acc4a0b0b364c1b2ace7ed210758355391f70b728e9a3a5186ae87d87"} Nov 02 22:32:45.271724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.269623 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd7b447acc4a0b0b364c1b2ace7ed210758355391f70b728e9a3a5186ae87d87" Nov 02 22:32:45.271724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.269726 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg" Nov 02 22:32:45.280238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.280198 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" event={"ID":"01763f54-474e-48c8-b743-3c3e2dfdfca9","Type":"ContainerStarted","Data":"21a828df79592611cd1bac0abd12bfda69935d43dfea84cd1bfd610034d3373b"} Nov 02 22:32:45.280513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.280493 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" event={"ID":"01763f54-474e-48c8-b743-3c3e2dfdfca9","Type":"ContainerStarted","Data":"287bc9ce3507808287c96fb71db647aa89c8176797e386cbc4e79cb43f4e688d"} Nov 02 22:32:45.287773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.287739 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" event={"ID":"8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe","Type":"ContainerDied","Data":"ad0678e1947287a9643e114a3f663ce30bbade0ad8de211c5c69cb9a93baac65"} Nov 02 22:32:45.289493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.289469 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad0678e1947287a9643e114a3f663ce30bbade0ad8de211c5c69cb9a93baac65" Nov 02 22:32:45.289812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.289787 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286" Nov 02 22:32:45.412103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.412032 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" podStartSLOduration=2.41200699 podStartE2EDuration="2.41200699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:45.402849709 +0000 UTC m=+1601.667842932" watchObservedRunningTime="2026-11-02 22:32:45.41200699 +0000 UTC m=+1601.677000210" Nov 02 22:32:45.895701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:45.895653 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd91f3cb-acc8-4005-871b-f9a4699bc8dd" path="/var/lib/kubelet/pods/cd91f3cb-acc8-4005-871b-f9a4699bc8dd/volumes" Nov 02 22:32:46.066323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.066276 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n"] Nov 02 22:32:46.070853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.070824 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf"] Nov 02 22:32:46.117275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.117238 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24"] Nov 02 22:32:46.122206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.122173 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2ab4dd-gbv4n"] Nov 02 22:32:46.126180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.126152 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29fde503-xf8nf"] Nov 02 22:32:46.129585 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.129547 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc7b724-d9f24"] Nov 02 22:32:46.132799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.132767 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95"] Nov 02 22:32:46.133435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.133421 5109 cpu_manager.go:414] "containerMap: removing container" podUID="8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" containerName="revoke-keys" Nov 02 22:32:46.133473 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.133437 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" containerName="revoke-keys" Nov 02 22:32:46.133509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.133476 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7ee07c24-deb2-4296-a3ad-c7bede1c7d92" containerName="revoke-keys" Nov 02 22:32:46.133509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.133482 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee07c24-deb2-4296-a3ad-c7bede1c7d92" containerName="revoke-keys" Nov 02 22:32:46.133599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.133588 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee07c24-deb2-4296-a3ad-c7bede1c7d92" containerName="revoke-keys" Nov 02 22:32:46.133659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.133604 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" containerName="revoke-keys" Nov 02 22:32:46.137346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.137330 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.171639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.171519 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95"] Nov 02 22:32:46.192670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.192623 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.192878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.192858 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpsvj\" (UniqueName: \"kubernetes.io/projected/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-kube-api-access-hpsvj\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.193231 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.193201 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.294219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.294184 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpsvj\" (UniqueName: \"kubernetes.io/projected/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-kube-api-access-hpsvj\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.294427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.294269 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.294427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.294401 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.295002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.294979 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.297004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.296982 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.298090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.298055 5109 generic.go:356] "Generic (PLEG): container finished" podID="01763f54-474e-48c8-b743-3c3e2dfdfca9" containerID="21a828df79592611cd1bac0abd12bfda69935d43dfea84cd1bfd610034d3373b" exitCode=0 Nov 02 22:32:46.298204 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.298105 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" event={"ID":"01763f54-474e-48c8-b743-3c3e2dfdfca9","Type":"ContainerDied","Data":"21a828df79592611cd1bac0abd12bfda69935d43dfea84cd1bfd610034d3373b"} Nov 02 22:32:46.355670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.355631 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpsvj\" (UniqueName: \"kubernetes.io/projected/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-kube-api-access-hpsvj\") pod \"maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.452061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.451980 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:46.682539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.682514 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:32:46.700791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.700757 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467202fa-5d84-41d6-b713-390961ca40f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467202fa-5d84-41d6-b713-390961ca40f8-istio-podinfo\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.700791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.700796 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-envoy\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.700998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.700841 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-data\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.700998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.700913 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-istio-token\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-istio-token\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.700998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.700956 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-credential-socket\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.701108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701056 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-kube-api-access-z852k\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-kube-api-access-z852k\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.701108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701090 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467202fa-5d84-41d6-b713-390961ca40f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467202fa-5d84-41d6-b713-390961ca40f8-istiod-ca-cert\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.701274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701220 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-data" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:46.701428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701275 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-credential-socket" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:46.701428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701278 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-socket\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.701428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701358 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-certs\") pod \"467202fa-5d84-41d6-b713-390961ca40f8\" (UID: \"467202fa-5d84-41d6-b713-390961ca40f8\") " Nov 02 22:32:46.701882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701507 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-socket" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:46.701882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701651 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467202fa-5d84-41d6-b713-390961ca40f8-istiod-ca-cert" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:46.702000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.701942 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-certs" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:46.702356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.702302 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.702356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.702333 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.702356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.702350 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467202fa-5d84-41d6-b713-390961ca40f8-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.702604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.702367 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.702604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.702382 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.703426 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.703401 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467202fa-5d84-41d6-b713-390961ca40f8-istio-podinfo" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:46.703510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.703475 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-istio-token" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:46.703549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.703526 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-kube-api-access-z852k" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "kube-api-access-z852k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:46.703637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.703616 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-envoy" pod "467202fa-5d84-41d6-b713-390961ca40f8" (UID: "467202fa-5d84-41d6-b713-390961ca40f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:46.804514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.804473 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.804514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.804508 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z852k\" (UniqueName: \"kubernetes.io/projected/467202fa-5d84-41d6-b713-390961ca40f8-kube-api-access-z852k\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.804514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.804523 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467202fa-5d84-41d6-b713-390961ca40f8-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.804822 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.804535 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467202fa-5d84-41d6-b713-390961ca40f8-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:46.833007 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:32:46.832976 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd823bfbc_36c0_486e_88b4_12e0b20e7ee3.slice/crio-50195d9fc61c5fa7b301acb1ae55abc5418289ad016ef309789c729bd6341d95 WatchSource:0}: Error finding container 50195d9fc61c5fa7b301acb1ae55abc5418289ad016ef309789c729bd6341d95: Status 404 returned error can't find the container with id 50195d9fc61c5fa7b301acb1ae55abc5418289ad016ef309789c729bd6341d95 Nov 02 22:32:46.837331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:46.837303 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95"] Nov 02 22:32:47.056083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.056046 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j"] Nov 02 22:32:47.062795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.062759 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395e0025-5jn8j"] Nov 02 22:32:47.104685 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.104655 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c"] Nov 02 22:32:47.204414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.204379 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g"] Nov 02 22:32:47.321777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.318999 5109 generic.go:356] "Generic (PLEG): container finished" podID="467202fa-5d84-41d6-b713-390961ca40f8" containerID="56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d" exitCode=0 Nov 02 22:32:47.321777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.319300 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" Nov 02 22:32:47.321777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.319452 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" event={"ID":"467202fa-5d84-41d6-b713-390961ca40f8","Type":"ContainerDied","Data":"56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d"} Nov 02 22:32:47.321777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.319615 5109 scope.go:122] "RemoveContainer" containerID="56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d" Nov 02 22:32:47.323161 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.323115 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8" event={"ID":"467202fa-5d84-41d6-b713-390961ca40f8","Type":"ContainerDied","Data":"26e4043beb03427148bae59f57aba88067f485ac9fa42ba4ad6cc4b368361295"} Nov 02 22:32:47.330439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.330402 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" event={"ID":"d823bfbc-36c0-486e-88b4-12e0b20e7ee3","Type":"ContainerStarted","Data":"a8fb90947179ae3c93c3b958df050178775f0777db2d416bb2272867617c5e43"} Nov 02 22:32:47.330589 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.330500 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" event={"ID":"d823bfbc-36c0-486e-88b4-12e0b20e7ee3","Type":"ContainerStarted","Data":"50195d9fc61c5fa7b301acb1ae55abc5418289ad016ef309789c729bd6341d95"} Nov 02 22:32:47.342210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.342192 5109 scope.go:122] "RemoveContainer" containerID="ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9" Nov 02 22:32:47.353076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.353051 5109 scope.go:122] "RemoveContainer" containerID="56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d" Nov 02 22:32:47.353448 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:47.353426 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d\": container with ID starting with 56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d not found: ID does not exist" containerID="56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d" Nov 02 22:32:47.353727 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.353463 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d"} err="failed to get container status \"56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d\": rpc error: code = NotFound desc = could not find container \"56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d\": container with ID starting with 56950e702962e0ac4be869711d524bad68a0bbc897d4ad245f93dcbc17ad245d not found: ID does not exist" Nov 02 22:32:47.353727 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.353488 5109 scope.go:122] "RemoveContainer" containerID="ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9" Nov 02 22:32:47.353869 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:47.353851 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9\": container with ID starting with ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9 not found: ID does not exist" containerID="ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9" Nov 02 22:32:47.353914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.353877 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9"} err="failed to get container status \"ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9\": rpc error: code = NotFound desc = could not find container \"ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9\": container with ID starting with ee5d9bb4fba478ed4e48073a1de311d66aa42600398e167cf45787c900398ea9 not found: ID does not exist" Nov 02 22:32:47.456761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.456706 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" podStartSLOduration=1.456686919 podStartE2EDuration="1.456686919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:47.454382422 +0000 UTC m=+1603.719375643" watchObservedRunningTime="2026-11-02 22:32:47.456686919 +0000 UTC m=+1603.721680138" Nov 02 22:32:47.633793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.628283 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8"] Nov 02 22:32:47.659002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.651322 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2663d62e-gw-openshift-default-5bc996dd49-vndd8"] Nov 02 22:32:47.674620 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.674450 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:47.759991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.757273 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01763f54-474e-48c8-b743-3c3e2dfdfca9-kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/01763f54-474e-48c8-b743-3c3e2dfdfca9-kube-api-access-ms6w5\") pod \"01763f54-474e-48c8-b743-3c3e2dfdfca9\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " Nov 02 22:32:47.759991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.757399 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-openshift-ca\") pod \"01763f54-474e-48c8-b743-3c3e2dfdfca9\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " Nov 02 22:32:47.759991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.757449 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-xks-ca\") pod \"01763f54-474e-48c8-b743-3c3e2dfdfca9\" (UID: \"01763f54-474e-48c8-b743-3c3e2dfdfca9\") " Nov 02 22:32:47.767606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.764366 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-openshift-ca" pod "01763f54-474e-48c8-b743-3c3e2dfdfca9" (UID: "01763f54-474e-48c8-b743-3c3e2dfdfca9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:47.778339 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.778121 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01763f54-474e-48c8-b743-3c3e2dfdfca9-kube-api-access-ms6w5" pod "01763f54-474e-48c8-b743-3c3e2dfdfca9" (UID: "01763f54-474e-48c8-b743-3c3e2dfdfca9"). InnerVolumeSpecName "kube-api-access-ms6w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:47.778708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.778688 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-xks-ca" pod "01763f54-474e-48c8-b743-3c3e2dfdfca9" (UID: "01763f54-474e-48c8-b743-3c3e2dfdfca9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:47.874864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.874785 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/01763f54-474e-48c8-b743-3c3e2dfdfca9-kube-api-access-ms6w5\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:47.874864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.874815 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:47.874864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.874829 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01763f54-474e-48c8-b743-3c3e2dfdfca9-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:47.894303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.894270 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3f0a39-4389-4545-8bfc-0545b2a0d293" path="/var/lib/kubelet/pods/2c3f0a39-4389-4545-8bfc-0545b2a0d293/volumes" Nov 02 22:32:47.894746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.894732 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467202fa-5d84-41d6-b713-390961ca40f8" path="/var/lib/kubelet/pods/467202fa-5d84-41d6-b713-390961ca40f8/volumes" Nov 02 22:32:47.895151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.895138 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99faa56c-3865-43a8-950b-86c32f17c370" path="/var/lib/kubelet/pods/99faa56c-3865-43a8-950b-86c32f17c370/volumes" Nov 02 22:32:47.895874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.895860 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9a787d-018e-4410-8638-e82beba29e77" path="/var/lib/kubelet/pods/da9a787d-018e-4410-8638-e82beba29e77/volumes" Nov 02 22:32:47.896278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:47.896252 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9039ba1-3275-4eb5-8bb8-756424870347" path="/var/lib/kubelet/pods/f9039ba1-3275-4eb5-8bb8-756424870347/volumes" Nov 02 22:32:48.340454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:48.340422 5109 generic.go:356] "Generic (PLEG): container finished" podID="d823bfbc-36c0-486e-88b4-12e0b20e7ee3" containerID="a8fb90947179ae3c93c3b958df050178775f0777db2d416bb2272867617c5e43" exitCode=0 Nov 02 22:32:48.340955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:48.340636 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" event={"ID":"d823bfbc-36c0-486e-88b4-12e0b20e7ee3","Type":"ContainerDied","Data":"a8fb90947179ae3c93c3b958df050178775f0777db2d416bb2272867617c5e43"} Nov 02 22:32:48.343257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:48.343238 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" Nov 02 22:32:48.343374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:48.343273 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz" event={"ID":"01763f54-474e-48c8-b743-3c3e2dfdfca9","Type":"ContainerDied","Data":"287bc9ce3507808287c96fb71db647aa89c8176797e386cbc4e79cb43f4e688d"} Nov 02 22:32:48.343374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:48.343312 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="287bc9ce3507808287c96fb71db647aa89c8176797e386cbc4e79cb43f4e688d" Nov 02 22:32:49.542956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.542932 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:49.602051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.602011 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-openshift-ca\") pod \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " Nov 02 22:32:49.602259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.602150 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-kube-api-access-hpsvj\" (UniqueName: \"kubernetes.io/projected/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-kube-api-access-hpsvj\") pod \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " Nov 02 22:32:49.602259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.602227 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-xks-ca\") pod \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\" (UID: \"d823bfbc-36c0-486e-88b4-12e0b20e7ee3\") " Nov 02 22:32:49.602504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.602475 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-openshift-ca" pod "d823bfbc-36c0-486e-88b4-12e0b20e7ee3" (UID: "d823bfbc-36c0-486e-88b4-12e0b20e7ee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:49.603012 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.602984 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:49.608199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.608168 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-xks-ca" pod "d823bfbc-36c0-486e-88b4-12e0b20e7ee3" (UID: "d823bfbc-36c0-486e-88b4-12e0b20e7ee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:49.610069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.610038 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-kube-api-access-hpsvj" pod "d823bfbc-36c0-486e-88b4-12e0b20e7ee3" (UID: "d823bfbc-36c0-486e-88b4-12e0b20e7ee3"). InnerVolumeSpecName "kube-api-access-hpsvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:49.632308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.632230 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp"] Nov 02 22:32:49.632475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.632455 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" containerID="cri-o://d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e" gracePeriod=30 Nov 02 22:32:49.639884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.639857 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm"] Nov 02 22:32:49.640054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.640036 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" containerID="cri-o://1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a" gracePeriod=30 Nov 02 22:32:49.680330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.680288 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z"] Nov 02 22:32:49.680551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.680505 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" containerID="cri-o://b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e" gracePeriod=30 Nov 02 22:32:49.685835 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.685799 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj"] Nov 02 22:32:49.688096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.687051 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" containerID="cri-o://bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2" gracePeriod=30 Nov 02 22:32:49.791591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.767069 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpsvj\" (UniqueName: \"kubernetes.io/projected/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-kube-api-access-hpsvj\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:49.791917 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:49.791894 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d823bfbc-36c0-486e-88b4-12e0b20e7ee3-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:50.380225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:50.380192 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" Nov 02 22:32:50.380392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:50.380226 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95" event={"ID":"d823bfbc-36c0-486e-88b4-12e0b20e7ee3","Type":"ContainerDied","Data":"50195d9fc61c5fa7b301acb1ae55abc5418289ad016ef309789c729bd6341d95"} Nov 02 22:32:50.380392 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:50.380277 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50195d9fc61c5fa7b301acb1ae55abc5418289ad016ef309789c729bd6341d95" Nov 02 22:32:50.530639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:50.530553 5109 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:32:50.530841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:50.530657 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:32:50.725233 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:50.724982 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72"] Nov 02 22:32:52.105909 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:52.105853 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" podUID="500893a0-a2f3-4840-b9ee-59e5226d9335" containerName="payload-processing" containerID="cri-o://021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151" gracePeriod=25 Nov 02 22:32:52.205373 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:52.205326 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" podUID="4da2ea58-8933-496e-a7d2-7711d131c55d" containerName="payload-pre-processing" containerID="cri-o://37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5" gracePeriod=25 Nov 02 22:32:53.049531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:53.049498 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc"] Nov 02 22:32:53.099164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:53.099125 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ada716-28mwc"] Nov 02 22:32:53.893251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:53.893208 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53060a2e-b077-4b08-a8db-704187f0e105" path="/var/lib/kubelet/pods/53060a2e-b077-4b08-a8db-704187f0e105/volumes" Nov 02 22:32:54.050180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:54.050144 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx"] Nov 02 22:32:54.100156 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:54.100109 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf43f723-vx8vx"] Nov 02 22:32:54.776658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:54.776626 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8"] Nov 02 22:32:54.845590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:54.845299 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" podUID="4da2ea58-8933-496e-a7d2-7711d131c55d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:32:54.847610 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:54.847552 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" podUID="500893a0-a2f3-4840-b9ee-59e5226d9335" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:32:54.891446 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:54.891410 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr"] Nov 02 22:32:55.055472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.055421 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv"] Nov 02 22:32:55.079599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.077878 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46f522-vxtkv"] Nov 02 22:32:55.156946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.156904 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:32:55.226380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.225438 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:32:55.250239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.250208 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:32:55.265437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.265407 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:32:55.293653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.288098 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-certs\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.293653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.288269 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istiod-ca-cert\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.293653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.288358 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-data\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.293653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.288458 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-podinfo\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.293653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.288536 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-token\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-token\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.293653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.288656 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-credential-socket\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.306655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.299379 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-data" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.306655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.299815 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-certs" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.306655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.300184 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istiod-ca-cert" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:55.306655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.302002 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-credential-socket" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.319647 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-kube-api-access-t8bjx\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-kube-api-access-t8bjx\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.319771 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-socket\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.319996 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-envoy\") pod \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\" (UID: \"615227d4-61d6-4f60-91b1-cc5a65a5bd4a\") " Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.320052 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-token\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-token\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.320902 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.320932 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.320948 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.324601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.320965 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.332807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.332720 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-socket" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.353076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.353005 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-envoy" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.361095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.361027 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-token" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.386841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.386777 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-token" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.386841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.386823 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-kube-api-access-t8bjx" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "kube-api-access-t8bjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.404596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.404535 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-podinfo" pod "615227d4-61d6-4f60-91b1-cc5a65a5bd4a" (UID: "615227d4-61d6-4f60-91b1-cc5a65a5bd4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:55.422447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422397 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-credential-socket\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.422634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422457 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-data\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.422634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422510 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-kube-api-access-ddgcr\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-kube-api-access-ddgcr\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.422634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422558 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-socket\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.422767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422672 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-kube-api-access-rzn2h\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-kube-api-access-rzn2h\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.422767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422715 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-data\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.422847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422774 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-credential-socket\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.422847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422779 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-data" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.422847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422810 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08b9afc-31fe-4677-82fd-0eb278017edf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08b9afc-31fe-4677-82fd-0eb278017edf-istiod-ca-cert\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.422847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422839 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-podinfo\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.423029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422869 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-token\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-token\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.423029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422902 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-podinfo\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.423029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.422947 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-data\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.423029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423003 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-token\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-token\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.423239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423041 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-envoy\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.423239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423087 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-credential-socket\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.423239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423130 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istiod-ca-cert\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.423239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423168 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-socket\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.423239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423213 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-podinfo\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.423239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423226 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-data" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.423558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423249 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-credential-socket" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.423558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423252 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-certs\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.423558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423300 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08b9afc-31fe-4677-82fd-0eb278017edf-istiod-ca-cert" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:55.423558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423319 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-envoy\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.423558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423422 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-kube-api-access-cxnxr\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-kube-api-access-cxnxr\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.423558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423520 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-socket\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.423558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423532 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-certs" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.423969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423585 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5ef48c-b4a6-4145-97aa-222763ba1453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5ef48c-b4a6-4145-97aa-222763ba1453-istiod-ca-cert\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.423969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423660 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-envoy\") pod \"c08b9afc-31fe-4677-82fd-0eb278017edf\" (UID: \"c08b9afc-31fe-4677-82fd-0eb278017edf\") " Nov 02 22:32:55.423969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423716 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-certs\") pod \"9c5ef48c-b4a6-4145-97aa-222763ba1453\" (UID: \"9c5ef48c-b4a6-4145-97aa-222763ba1453\") " Nov 02 22:32:55.423969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423720 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-socket" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.423969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423729 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-credential-socket" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.423969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423827 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-certs\") pod \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\" (UID: \"eaf2116c-cb61-4b3e-a6dd-e633a53d254b\") " Nov 02 22:32:55.424308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.423984 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-socket" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.424432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424379 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5ef48c-b4a6-4145-97aa-222763ba1453-istiod-ca-cert" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:55.424538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424510 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istiod-ca-cert" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:55.424657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424636 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-certs" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.424841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424813 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.424930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424850 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.424930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424866 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.424930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424881 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.424930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424895 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.424930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424910 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.424930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424924 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424938 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08b9afc-31fe-4677-82fd-0eb278017edf-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424953 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424966 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424980 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.424995 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.425010 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bjx\" (UniqueName: \"kubernetes.io/projected/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-kube-api-access-t8bjx\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.425027 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.425043 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615227d4-61d6-4f60-91b1-cc5a65a5bd4a-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.425058 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.425072 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5ef48c-b4a6-4145-97aa-222763ba1453-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.425456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.425297 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-credential-socket" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.425507 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.425482 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-certs" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.427118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.427093 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-socket" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.430216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.430049 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-kube-api-access-rzn2h" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "kube-api-access-rzn2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.430216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.430090 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-token" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.430397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.430226 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-data" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.436688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.436658 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-podinfo" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:55.440004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.439971 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-podinfo" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:55.446151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.446099 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-envoy" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.446151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.446124 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-kube-api-access-cxnxr" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "kube-api-access-cxnxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.446420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.446160 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-token" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.446694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.446584 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-envoy" pod "eaf2116c-cb61-4b3e-a6dd-e633a53d254b" (UID: "eaf2116c-cb61-4b3e-a6dd-e633a53d254b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.446694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.446638 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-envoy" pod "9c5ef48c-b4a6-4145-97aa-222763ba1453" (UID: "9c5ef48c-b4a6-4145-97aa-222763ba1453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.446694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.446659 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-kube-api-access-ddgcr" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "kube-api-access-ddgcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.449317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.449277 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-podinfo" pod "c08b9afc-31fe-4677-82fd-0eb278017edf" (UID: "c08b9afc-31fe-4677-82fd-0eb278017edf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:55.450039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.450012 5109 generic.go:356] "Generic (PLEG): container finished" podID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerID="d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e" exitCode=0 Nov 02 22:32:55.450140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.450070 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" Nov 02 22:32:55.450140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.450099 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" event={"ID":"9c5ef48c-b4a6-4145-97aa-222763ba1453","Type":"ContainerDied","Data":"d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e"} Nov 02 22:32:55.450253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.450169 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp" event={"ID":"9c5ef48c-b4a6-4145-97aa-222763ba1453","Type":"ContainerDied","Data":"753cae0b36b547fe37888ed82c288c7091df582419296f7eca174b0732750460"} Nov 02 22:32:55.450253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.450203 5109 scope.go:122] "RemoveContainer" containerID="d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e" Nov 02 22:32:55.453544 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.453519 5109 generic.go:356] "Generic (PLEG): container finished" podID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerID="b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e" exitCode=0 Nov 02 22:32:55.453706 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.453560 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" event={"ID":"c08b9afc-31fe-4677-82fd-0eb278017edf","Type":"ContainerDied","Data":"b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e"} Nov 02 22:32:55.453706 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.453696 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" Nov 02 22:32:55.453823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.453698 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z" event={"ID":"c08b9afc-31fe-4677-82fd-0eb278017edf","Type":"ContainerDied","Data":"3ab9ce912c3a699b48920895cda45fa039cb9974a3ea31412affd4dba85ddd5d"} Nov 02 22:32:55.455728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.455701 5109 generic.go:356] "Generic (PLEG): container finished" podID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerID="1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a" exitCode=0 Nov 02 22:32:55.455853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.455775 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" event={"ID":"615227d4-61d6-4f60-91b1-cc5a65a5bd4a","Type":"ContainerDied","Data":"1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a"} Nov 02 22:32:55.455853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.455829 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" event={"ID":"615227d4-61d6-4f60-91b1-cc5a65a5bd4a","Type":"ContainerDied","Data":"382b2bae61a928a5c9107d0878632c5da3be9be85d4e28dcdf0797890d6c469c"} Nov 02 22:32:55.455971 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.455878 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm" Nov 02 22:32:55.458006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.457959 5109 generic.go:356] "Generic (PLEG): container finished" podID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerID="bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2" exitCode=0 Nov 02 22:32:55.458114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.458018 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" event={"ID":"eaf2116c-cb61-4b3e-a6dd-e633a53d254b","Type":"ContainerDied","Data":"bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2"} Nov 02 22:32:55.458114 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.458075 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" event={"ID":"eaf2116c-cb61-4b3e-a6dd-e633a53d254b","Type":"ContainerDied","Data":"607c7d85b44905e3d3f10de95f57a1947fdb992cc302f11411ec99567a92ae0c"} Nov 02 22:32:55.458234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.458164 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj" Nov 02 22:32:55.473804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.473777 5109 scope.go:122] "RemoveContainer" containerID="d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e" Nov 02 22:32:55.474194 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:55.474131 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e\": container with ID starting with d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e not found: ID does not exist" containerID="d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e" Nov 02 22:32:55.474194 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.474171 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e"} err="failed to get container status \"d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e\": rpc error: code = NotFound desc = could not find container \"d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e\": container with ID starting with d6a265a78c0d2cfe65c96af49a420439ee12b5d9e3b38843c40b88ac9e229a5e not found: ID does not exist" Nov 02 22:32:55.474363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.474195 5109 scope.go:122] "RemoveContainer" containerID="b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e" Nov 02 22:32:55.490408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.490386 5109 scope.go:122] "RemoveContainer" containerID="b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e" Nov 02 22:32:55.490809 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:55.490782 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e\": container with ID starting with b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e not found: ID does not exist" containerID="b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e" Nov 02 22:32:55.490942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.490816 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e"} err="failed to get container status \"b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e\": rpc error: code = NotFound desc = could not find container \"b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e\": container with ID starting with b9be01df97576b08bf92f2807f306c884f1756fccf4654711ca24727fa7bdf7e not found: ID does not exist" Nov 02 22:32:55.490942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.490840 5109 scope.go:122] "RemoveContainer" containerID="1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a" Nov 02 22:32:55.506619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.505171 5109 scope.go:122] "RemoveContainer" containerID="1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a" Nov 02 22:32:55.506619 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:55.505557 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a\": container with ID starting with 1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a not found: ID does not exist" containerID="1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a" Nov 02 22:32:55.506619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.505610 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a"} err="failed to get container status \"1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a\": rpc error: code = NotFound desc = could not find container \"1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a\": container with ID starting with 1f72c84f5fc9e4076381e0b11e1492af6cc356af7942eaeeac31358617c5536a not found: ID does not exist" Nov 02 22:32:55.506619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.505637 5109 scope.go:122] "RemoveContainer" containerID="bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2" Nov 02 22:32:55.517524 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.517506 5109 scope.go:122] "RemoveContainer" containerID="bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2" Nov 02 22:32:55.517890 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:55.517856 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2\": container with ID starting with bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2 not found: ID does not exist" containerID="bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2" Nov 02 22:32:55.517947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.517903 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2"} err="failed to get container status \"bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2\": rpc error: code = NotFound desc = could not find container \"bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2\": container with ID starting with bc929ccc801d4102b8957bc5bf503b06cc00eea34291b9c3110275165c0594f2 not found: ID does not exist" Nov 02 22:32:55.526371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526347 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526373 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526383 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxnxr\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-kube-api-access-cxnxr\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526398 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526412 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08b9afc-31fe-4677-82fd-0eb278017edf-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526422 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526432 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddgcr\" (UniqueName: \"kubernetes.io/projected/c08b9afc-31fe-4677-82fd-0eb278017edf-kube-api-access-ddgcr\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526441 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzn2h\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-kube-api-access-rzn2h\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526449 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526457 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526465 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526479 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526491 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526499 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5ef48c-b4a6-4145-97aa-222763ba1453-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.526503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.526507 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf2116c-cb61-4b3e-a6dd-e633a53d254b-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.771701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.771661 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" containerID="cri-o://091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3" gracePeriod=45 Nov 02 22:32:55.793782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.793754 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp"] Nov 02 22:32:55.797914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.797892 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-hzprp"] Nov 02 22:32:55.893582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.893546 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c4b4e5-562e-49b0-b0d4-7cb4dd76013c" path="/var/lib/kubelet/pods/02c4b4e5-562e-49b0-b0d4-7cb4dd76013c/volumes" Nov 02 22:32:55.894112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.894096 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6e68c5-3cd1-4202-a238-e10d85c61a6d" path="/var/lib/kubelet/pods/0b6e68c5-3cd1-4202-a238-e10d85c61a6d/volumes" Nov 02 22:32:55.894634 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.894618 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" path="/var/lib/kubelet/pods/9c5ef48c-b4a6-4145-97aa-222763ba1453/volumes" Nov 02 22:32:55.913004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.912925 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm"] Nov 02 22:32:55.918795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:55.918764 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-t5tnm"] Nov 02 22:32:56.025378 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.025330 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z"] Nov 02 22:32:56.029729 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.029690 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-z5k6z"] Nov 02 22:32:56.067784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.067760 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:32:56.139100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.139045 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj"] Nov 02 22:32:56.142987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.142952 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-qh4kj"] Nov 02 22:32:56.271654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.245389 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-model-cache\") pod \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " Nov 02 22:32:56.272978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.271970 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tls-certs\") pod \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " Nov 02 22:32:56.273383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.273352 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tmp-dir\") pod \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " Nov 02 22:32:56.279625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.273672 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-home\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-home\") pod \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " Nov 02 22:32:56.279868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.279839 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-kube-api-access-45rpx\" (UniqueName: \"kubernetes.io/projected/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-kube-api-access-45rpx\") pod \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " Nov 02 22:32:56.285705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.285674 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-dshm\") pod \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\" (UID: \"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537\") " Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.345007 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tmp-dir" pod "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" (UID: "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.347082 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-model-cache" pod "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" (UID: "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.352776 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-home" pod "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" (UID: "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.381297 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tls-certs" pod "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" (UID: "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.400976 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-kube-api-access-45rpx" pod "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" (UID: "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537"). InnerVolumeSpecName "kube-api-access-45rpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.402226 5109 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-model-cache\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.402257 5109 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tls-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.402271 5109 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-tmp-dir\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.402284 5109 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-home\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.424405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.402299 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rpx\" (UniqueName: \"kubernetes.io/projected/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-kube-api-access-45rpx\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.455917 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.455884 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-dshm" pod "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" (UID: "ed6c83a3-6d80-4d04-a20f-27a5e8a7e537"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.471261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.471226 5109 generic.go:356] "Generic (PLEG): container finished" podID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerID="091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3" exitCode=0 Nov 02 22:32:56.471413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.471354 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" event={"ID":"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537","Type":"ContainerDied","Data":"091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3"} Nov 02 22:32:56.471413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.471392 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" event={"ID":"ed6c83a3-6d80-4d04-a20f-27a5e8a7e537","Type":"ContainerDied","Data":"4ee8188772d6bdce6d441c3df167c921b6cf2d55a4ff0cc40080ca3698eae554"} Nov 02 22:32:56.471492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.471414 5109 scope.go:122] "RemoveContainer" containerID="091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3" Nov 02 22:32:56.471638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.471543 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf" Nov 02 22:32:56.507160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.507126 5109 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537-dshm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.509893 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:56.509847 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:32:56.510763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.510742 5109 scope.go:122] "RemoveContainer" containerID="091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3" Nov 02 22:32:56.511542 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:32:56.511512 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3\": container with ID starting with 091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3 not found: ID does not exist" containerID="091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3" Nov 02 22:32:56.512019 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.511548 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3"} err="failed to get container status \"091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3\": rpc error: code = NotFound desc = could not find container \"091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3\": container with ID starting with 091d58ce99a10038f0fdcac03da563c4b591d16cf7db03bf4bb7ff999d8d33d3 not found: ID does not exist" Nov 02 22:32:56.671077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.666656 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf"] Nov 02 22:32:56.686048 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:56.686010 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/distinct-w6-49a557-kserve-6bc6b5694-rr5wf"] Nov 02 22:32:57.137508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.137471 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4"] Nov 02 22:32:57.239738 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.239705 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc"] Nov 02 22:32:57.824722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.824685 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln"] Nov 02 22:32:57.824976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.824936 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" containerID="cri-o://8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3" gracePeriod=30 Nov 02 22:32:57.830070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.830043 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d"] Nov 02 22:32:57.830277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.830253 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" containerID="cri-o://969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3" gracePeriod=30 Nov 02 22:32:57.834676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.834636 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6"] Nov 02 22:32:57.834933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.834905 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" containerID="cri-o://c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882" gracePeriod=30 Nov 02 22:32:57.838314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.838287 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75"] Nov 02 22:32:57.838468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.838448 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" containerID="cri-o://6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b" gracePeriod=30 Nov 02 22:32:57.893844 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.893811 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" path="/var/lib/kubelet/pods/615227d4-61d6-4f60-91b1-cc5a65a5bd4a/volumes" Nov 02 22:32:57.894344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.894330 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" path="/var/lib/kubelet/pods/c08b9afc-31fe-4677-82fd-0eb278017edf/volumes" Nov 02 22:32:57.894826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.894811 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" path="/var/lib/kubelet/pods/eaf2116c-cb61-4b3e-a6dd-e633a53d254b/volumes" Nov 02 22:32:57.895549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:57.895533 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" path="/var/lib/kubelet/pods/ed6c83a3-6d80-4d04-a20f-27a5e8a7e537/volumes" Nov 02 22:32:59.368758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.368724 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8"] Nov 02 22:32:59.369348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369335 5109 cpu_manager.go:414] "containerMap: removing container" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" Nov 02 22:32:59.369398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369349 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" Nov 02 22:32:59.369398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369365 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" Nov 02 22:32:59.369398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369370 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" Nov 02 22:32:59.369398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369389 5109 cpu_manager.go:414] "containerMap: removing container" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" Nov 02 22:32:59.369398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369394 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369407 5109 cpu_manager.go:414] "containerMap: removing container" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369413 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369420 5109 cpu_manager.go:414] "containerMap: removing container" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369424 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369443 5109 cpu_manager.go:414] "containerMap: removing container" podUID="01763f54-474e-48c8-b743-3c3e2dfdfca9" containerName="revoke-keys" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369447 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01763f54-474e-48c8-b743-3c3e2dfdfca9" containerName="revoke-keys" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369456 5109 cpu_manager.go:414] "containerMap: removing container" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369461 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369466 5109 cpu_manager.go:414] "containerMap: removing container" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369471 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369482 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d823bfbc-36c0-486e-88b4-12e0b20e7ee3" containerName="revoke-keys" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369487 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d823bfbc-36c0-486e-88b4-12e0b20e7ee3" containerName="revoke-keys" Nov 02 22:32:59.369559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369584 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369592 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="467202fa-5d84-41d6-b713-390961ca40f8" containerName="istio-proxy" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369603 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="01763f54-474e-48c8-b743-3c3e2dfdfca9" containerName="revoke-keys" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369610 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5ef48c-b4a6-4145-97aa-222763ba1453" containerName="istio-proxy" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369616 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf2116c-cb61-4b3e-a6dd-e633a53d254b" containerName="istio-proxy" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369621 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d823bfbc-36c0-486e-88b4-12e0b20e7ee3" containerName="revoke-keys" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369627 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6c83a3-6d80-4d04-a20f-27a5e8a7e537" containerName="main" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369633 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08b9afc-31fe-4677-82fd-0eb278017edf" containerName="istio-proxy" Nov 02 22:32:59.369998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.369641 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="615227d4-61d6-4f60-91b1-cc5a65a5bd4a" containerName="istio-proxy" Nov 02 22:32:59.374122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.374108 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.428018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.427979 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8"] Nov 02 22:32:59.433353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.433318 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4"] Nov 02 22:32:59.438893 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.438869 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.473274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.473244 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:32:59.473441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.473344 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:32:59.490523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.490495 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mds9k\" (UniqueName: \"kubernetes.io/projected/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-kube-api-access-mds9k\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.490722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.490701 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.490812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.490794 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.529813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.529779 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4"] Nov 02 22:32:59.591838 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.591802 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.592054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.591893 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.592054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.591928 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.592054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.591983 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.592243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.592144 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mds9k\" (UniqueName: \"kubernetes.io/projected/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-kube-api-access-mds9k\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.592243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.592236 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kspgs\" (UniqueName: \"kubernetes.io/projected/658ea0ef-0203-4ad7-982a-274f23cd6d37-kube-api-access-kspgs\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.592796 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.592775 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.594499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.594475 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.595810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.595789 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9"] Nov 02 22:32:59.600452 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.600432 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.658640 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.658490 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9"] Nov 02 22:32:59.659897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.659864 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mds9k\" (UniqueName: \"kubernetes.io/projected/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-kube-api-access-mds9k\") pod \"maas-api-revoke-sub-51342356-s9kc8\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.688773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.688730 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:32:59.698097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.698047 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.698268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.698253 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.698337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.698311 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.698443 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.698421 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.698543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.698463 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rl4\" (UniqueName: \"kubernetes.io/projected/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-kube-api-access-28rl4\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.698684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.698629 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kspgs\" (UniqueName: \"kubernetes.io/projected/658ea0ef-0203-4ad7-982a-274f23cd6d37-kube-api-access-kspgs\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.699073 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.699050 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.701029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.700996 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.761169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.761102 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kspgs\" (UniqueName: \"kubernetes.io/projected/658ea0ef-0203-4ad7-982a-274f23cd6d37-kube-api-access-kspgs\") pod \"maas-api-revoke-sub-70d300a2-vn2w4\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:32:59.778058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.777845 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" podUID="353686ce-7828-478a-b4ad-9f27c3fdc577" containerName="payload-processing" containerID="cri-o://b3b836ee829d9e345cdb67d4fe1918f37764e50313f4039608bf6ae147fb7a38" gracePeriod=25 Nov 02 22:32:59.800393 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.800354 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.800393 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.800408 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rl4\" (UniqueName: \"kubernetes.io/projected/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-kube-api-access-28rl4\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.800690 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.800502 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.802293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.802079 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.803712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.803691 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.825852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.825822 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc"] Nov 02 22:32:59.831924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.831895 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:32:59.862939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.862910 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rl4\" (UniqueName: \"kubernetes.io/projected/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-kube-api-access-28rl4\") pod \"maas-api-revoke-sub-21cb776d-xrgr9\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:32:59.893021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.892977 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" podUID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" containerName="payload-pre-processing" containerID="cri-o://613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029" gracePeriod=25 Nov 02 22:32:59.897152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.897122 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc"] Nov 02 22:32:59.920593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:32:59.920500 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:33:00.031771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.031737 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ltc\" (UniqueName: \"kubernetes.io/projected/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-kube-api-access-h2ltc\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.031882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.031819 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.032019 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.032002 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.056094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.056026 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:00.056265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.056121 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:00.057854 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.057828 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:33:00.077610 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.077129 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8"] Nov 02 22:33:00.079454 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:33:00.079406 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01b4628_5e7f_486a_9dd2_b5d47cf8df81.slice/crio-e56fadbbb8528c9afda1933b0f083bad0e55e8b2331eca4d9870f074b8ed88c5 WatchSource:0}: Error finding container e56fadbbb8528c9afda1933b0f083bad0e55e8b2331eca4d9870f074b8ed88c5: Status 404 returned error can't find the container with id e56fadbbb8528c9afda1933b0f083bad0e55e8b2331eca4d9870f074b8ed88c5 Nov 02 22:33:00.134920 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.134884 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ltc\" (UniqueName: \"kubernetes.io/projected/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-kube-api-access-h2ltc\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.135059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.134960 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.135154 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.135131 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.137273 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.136269 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.140014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.139990 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.202067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.201950 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ltc\" (UniqueName: \"kubernetes.io/projected/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-kube-api-access-h2ltc\") pod \"maas-api-revoke-sub-074e9677-87msc\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.472659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.447093 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9"] Nov 02 22:33:00.488895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.476828 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:00.500358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.495114 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4"] Nov 02 22:33:00.564043 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.563165 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" event={"ID":"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690","Type":"ContainerStarted","Data":"6ab33b7f8f5256e4b3e69a71afa7afe5270ec1fb1704b29af68cbdc1a450b58a"} Nov 02 22:33:00.603594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.574212 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" event={"ID":"c01b4628-5e7f-486a-9dd2-b5d47cf8df81","Type":"ContainerStarted","Data":"e56fadbbb8528c9afda1933b0f083bad0e55e8b2331eca4d9870f074b8ed88c5"} Nov 02 22:33:00.603594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.577310 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" event={"ID":"658ea0ef-0203-4ad7-982a-274f23cd6d37","Type":"ContainerStarted","Data":"f384fcb0db6e69173314c28b65f0d1e9b83d155ee94d94b3ca50426875ccd971"} Nov 02 22:33:00.874439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.874395 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" podUID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:00.874671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:00.874527 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" podUID="353686ce-7828-478a-b4ad-9f27c3fdc577" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:01.080618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.079681 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc"] Nov 02 22:33:01.587674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.587631 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" event={"ID":"658ea0ef-0203-4ad7-982a-274f23cd6d37","Type":"ContainerStarted","Data":"f2482fcc22ac91fd48556f964a8c10c1830284cca2da4e6aad42f9e6927d4baf"} Nov 02 22:33:01.589608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.589547 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" event={"ID":"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3","Type":"ContainerStarted","Data":"6289035c83b4e99421c64bd78e6605141206a519d2d51854fb4948d04e868311"} Nov 02 22:33:01.589759 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.589632 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" event={"ID":"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3","Type":"ContainerStarted","Data":"fd63c41590ef833b5f31e145fc360106bcf444062d92efb2e40453c8bd07099b"} Nov 02 22:33:01.591265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.591235 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" event={"ID":"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690","Type":"ContainerStarted","Data":"7cefc66e78c2e5109f470c20209e5730fd4cf9b6597999be0b401dc2ec1324b4"} Nov 02 22:33:01.593052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.593029 5109 generic.go:356] "Generic (PLEG): container finished" podID="c01b4628-5e7f-486a-9dd2-b5d47cf8df81" containerID="813c26b2d8099449bef43e1bfddfab24f3692641c96a89185827500111380f6a" exitCode=0 Nov 02 22:33:01.593131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.593073 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" event={"ID":"c01b4628-5e7f-486a-9dd2-b5d47cf8df81","Type":"ContainerDied","Data":"813c26b2d8099449bef43e1bfddfab24f3692641c96a89185827500111380f6a"} Nov 02 22:33:01.704549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.704495 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" podStartSLOduration=2.704476589 podStartE2EDuration="2.704476589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:33:01.703079275 +0000 UTC m=+1617.968072496" watchObservedRunningTime="2026-11-02 22:33:01.704476589 +0000 UTC m=+1617.969469807" Nov 02 22:33:01.828184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:01.828115 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" podStartSLOduration=2.828099479 podStartE2EDuration="2.828099479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:33:01.823611436 +0000 UTC m=+1618.088604646" watchObservedRunningTime="2026-11-02 22:33:01.828099479 +0000 UTC m=+1618.093092698" Nov 02 22:33:02.080593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.078766 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" podStartSLOduration=3.078744962 podStartE2EDuration="3.078744962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:33:02.076864535 +0000 UTC m=+1618.341857782" watchObservedRunningTime="2026-11-02 22:33:02.078744962 +0000 UTC m=+1618.343738182" Nov 02 22:33:02.138730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.138677 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" podUID="5a4b4ea2-9005-4fcf-9de2-29a19050ca73" containerName="payload-processing" containerID="cri-o://7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49" gracePeriod=25 Nov 02 22:33:02.170639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.170600 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x"] Nov 02 22:33:02.174184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.174155 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca85db-zv69x"] Nov 02 22:33:02.240933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.240893 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" podUID="57460eb1-1c58-44dd-b7a8-b2f9ad12e627" containerName="payload-pre-processing" containerID="cri-o://b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0" gracePeriod=25 Nov 02 22:33:02.400455 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.398968 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:33:02.419110 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.419065 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-plugins-config-volume\") pod \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " Nov 02 22:33:02.419305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.419146 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-kube-api-access-927w7\" (UniqueName: \"kubernetes.io/projected/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-kube-api-access-927w7\") pod \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\" (UID: \"5a4b4ea2-9005-4fcf-9de2-29a19050ca73\") " Nov 02 22:33:02.419598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.419540 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-plugins-config-volume" pod "5a4b4ea2-9005-4fcf-9de2-29a19050ca73" (UID: "5a4b4ea2-9005-4fcf-9de2-29a19050ca73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:02.419963 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.419937 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.422804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.422754 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-kube-api-access-927w7" pod "5a4b4ea2-9005-4fcf-9de2-29a19050ca73" (UID: "5a4b4ea2-9005-4fcf-9de2-29a19050ca73"). InnerVolumeSpecName "kube-api-access-927w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:02.531214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.527264 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927w7\" (UniqueName: \"kubernetes.io/projected/5a4b4ea2-9005-4fcf-9de2-29a19050ca73-kube-api-access-927w7\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.557782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.557750 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:33:02.605493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.605401 5109 generic.go:356] "Generic (PLEG): container finished" podID="5a4b4ea2-9005-4fcf-9de2-29a19050ca73" containerID="7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49" exitCode=0 Nov 02 22:33:02.605493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.605462 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" Nov 02 22:33:02.606011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.605495 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" event={"ID":"5a4b4ea2-9005-4fcf-9de2-29a19050ca73","Type":"ContainerDied","Data":"7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49"} Nov 02 22:33:02.606011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.605593 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4" event={"ID":"5a4b4ea2-9005-4fcf-9de2-29a19050ca73","Type":"ContainerDied","Data":"a76c89dd8398eb4f2b0e0bea29e7552ee2649ce6a8b14f81ceee63f3758928e0"} Nov 02 22:33:02.606011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.605628 5109 scope.go:122] "RemoveContainer" containerID="7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49" Nov 02 22:33:02.608023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.607994 5109 generic.go:356] "Generic (PLEG): container finished" podID="57460eb1-1c58-44dd-b7a8-b2f9ad12e627" containerID="b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0" exitCode=0 Nov 02 22:33:02.608348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.608062 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" Nov 02 22:33:02.608348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.608080 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" event={"ID":"57460eb1-1c58-44dd-b7a8-b2f9ad12e627","Type":"ContainerDied","Data":"b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0"} Nov 02 22:33:02.608348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.608138 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc" event={"ID":"57460eb1-1c58-44dd-b7a8-b2f9ad12e627","Type":"ContainerDied","Data":"77fdceeeb6dc5444dc8cd1dcb8c589b80cf98e0e69bfdb65c7768df62a662e3c"} Nov 02 22:33:02.613472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.613422 5109 generic.go:356] "Generic (PLEG): container finished" podID="f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" containerID="6289035c83b4e99421c64bd78e6605141206a519d2d51854fb4948d04e868311" exitCode=0 Nov 02 22:33:02.613645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.613486 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" event={"ID":"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3","Type":"ContainerDied","Data":"6289035c83b4e99421c64bd78e6605141206a519d2d51854fb4948d04e868311"} Nov 02 22:33:02.615907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.615885 5109 generic.go:356] "Generic (PLEG): container finished" podID="1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" containerID="7cefc66e78c2e5109f470c20209e5730fd4cf9b6597999be0b401dc2ec1324b4" exitCode=0 Nov 02 22:33:02.616021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.615965 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" event={"ID":"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690","Type":"ContainerDied","Data":"7cefc66e78c2e5109f470c20209e5730fd4cf9b6597999be0b401dc2ec1324b4"} Nov 02 22:33:02.618002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.617976 5109 generic.go:356] "Generic (PLEG): container finished" podID="658ea0ef-0203-4ad7-982a-274f23cd6d37" containerID="f2482fcc22ac91fd48556f964a8c10c1830284cca2da4e6aad42f9e6927d4baf" exitCode=0 Nov 02 22:33:02.618122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.618047 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" event={"ID":"658ea0ef-0203-4ad7-982a-274f23cd6d37","Type":"ContainerDied","Data":"f2482fcc22ac91fd48556f964a8c10c1830284cca2da4e6aad42f9e6927d4baf"} Nov 02 22:33:02.628705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.628673 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-plugins-config-volume\") pod \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " Nov 02 22:33:02.629215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.629037 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-plugins-config-volume" pod "57460eb1-1c58-44dd-b7a8-b2f9ad12e627" (UID: "57460eb1-1c58-44dd-b7a8-b2f9ad12e627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:02.629292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.629278 5109 scope.go:122] "RemoveContainer" containerID="7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49" Nov 02 22:33:02.629762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.629741 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-kube-api-access-blcf7\" (UniqueName: \"kubernetes.io/projected/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-kube-api-access-blcf7\") pod \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\" (UID: \"57460eb1-1c58-44dd-b7a8-b2f9ad12e627\") " Nov 02 22:33:02.629879 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:02.629819 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49\": container with ID starting with 7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49 not found: ID does not exist" containerID="7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49" Nov 02 22:33:02.629879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.629862 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49"} err="failed to get container status \"7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49\": rpc error: code = NotFound desc = could not find container \"7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49\": container with ID starting with 7e375e5e119adeb3a08002c3d225ad841536801a15d3697f597b6678c9e32b49 not found: ID does not exist" Nov 02 22:33:02.630013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.629888 5109 scope.go:122] "RemoveContainer" containerID="b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0" Nov 02 22:33:02.632453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.632422 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-kube-api-access-blcf7" pod "57460eb1-1c58-44dd-b7a8-b2f9ad12e627" (UID: "57460eb1-1c58-44dd-b7a8-b2f9ad12e627"). InnerVolumeSpecName "kube-api-access-blcf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:02.632656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.632636 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.660885 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.660441 5109 scope.go:122] "RemoveContainer" containerID="b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0" Nov 02 22:33:02.661230 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:02.661151 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0\": container with ID starting with b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0 not found: ID does not exist" containerID="b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0" Nov 02 22:33:02.661230 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.661187 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0"} err="failed to get container status \"b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0\": rpc error: code = NotFound desc = could not find container \"b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0\": container with ID starting with b7bba2920cc71d12bc5a50cb428f5fcd79fc2ca019ad3f6a7150013af318c9f0 not found: ID does not exist" Nov 02 22:33:02.745193 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.745141 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blcf7\" (UniqueName: \"kubernetes.io/projected/57460eb1-1c58-44dd-b7a8-b2f9ad12e627-kube-api-access-blcf7\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.777180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.777139 5109 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:02.777505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.777467 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:02.834907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.834879 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:33:02.947626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.947598 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-openshift-ca\") pod \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " Nov 02 22:33:02.947728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.947716 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-xks-ca\") pod \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " Nov 02 22:33:02.947926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.947759 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-kube-api-access-mds9k\" (UniqueName: \"kubernetes.io/projected/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-kube-api-access-mds9k\") pod \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\" (UID: \"c01b4628-5e7f-486a-9dd2-b5d47cf8df81\") " Nov 02 22:33:02.948060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.948033 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-openshift-ca" pod "c01b4628-5e7f-486a-9dd2-b5d47cf8df81" (UID: "c01b4628-5e7f-486a-9dd2-b5d47cf8df81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:02.948439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.948417 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.950888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.950860 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-kube-api-access-mds9k" pod "c01b4628-5e7f-486a-9dd2-b5d47cf8df81" (UID: "c01b4628-5e7f-486a-9dd2-b5d47cf8df81"). InnerVolumeSpecName "kube-api-access-mds9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:02.951093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:02.951063 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-xks-ca" pod "c01b4628-5e7f-486a-9dd2-b5d47cf8df81" (UID: "c01b4628-5e7f-486a-9dd2-b5d47cf8df81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:03.052817 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.052201 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.052817 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.052236 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mds9k\" (UniqueName: \"kubernetes.io/projected/c01b4628-5e7f-486a-9dd2-b5d47cf8df81-kube-api-access-mds9k\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.116220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.116159 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4"] Nov 02 22:33:03.123394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.121865 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98a222-55fcb5f7b5-25wz4"] Nov 02 22:33:03.146600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.146548 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:33:03.185731 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.185629 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:33:03.197923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.197513 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:33:03.229541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.229513 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:33:03.256149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256014 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a45c2e-274a-4f64-acf1-7836436b6c50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a45c2e-274a-4f64-acf1-7836436b6c50-istiod-ca-cert\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.256332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256238 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-istio-token\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-istio-token\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.256484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256458 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a45c2e-274a-4f64-acf1-7836436b6c50-istiod-ca-cert" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:03.256560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256511 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-kube-api-access-45v4x\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-kube-api-access-45v4x\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.256657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256555 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-envoy\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.256657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256635 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-certs\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.256767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256687 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-token\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-token\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.256767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256741 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-data\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.256866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256788 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-data\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.256866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256834 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb518472-d243-4a01-aead-146570bff41f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb518472-d243-4a01-aead-146570bff41f-istio-podinfo\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.256968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.256868 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fe2fec-540b-473f-8303-46752a067bdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fe2fec-540b-473f-8303-46752a067bdc-istiod-ca-cert\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.257045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257006 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-certs" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.257045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257021 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-credential-socket\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.257173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257070 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-envoy\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.257173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257133 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9fe2fec-540b-473f-8303-46752a067bdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fe2fec-540b-473f-8303-46752a067bdc-istio-podinfo\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.257316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257180 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-certs\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.257316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257230 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-kube-api-access-6w4t2\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-kube-api-access-6w4t2\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.257316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257248 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-data" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.257316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257281 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-socket\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257318 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-socket\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257419 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-kube-api-access-6x8xq\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257615 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-istio-token\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-istio-token\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257655 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb518472-d243-4a01-aead-146570bff41f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb518472-d243-4a01-aead-146570bff41f-istiod-ca-cert\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257695 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-certs\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257781 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-credential-socket\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257817 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-socket\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257854 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-podinfo\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257902 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-envoy\") pod \"e9fe2fec-540b-473f-8303-46752a067bdc\" (UID: \"e9fe2fec-540b-473f-8303-46752a067bdc\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.257954 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-data\") pod \"fb518472-d243-4a01-aead-146570bff41f\" (UID: \"fb518472-d243-4a01-aead-146570bff41f\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258005 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-credential-socket\") pod \"39a45c2e-274a-4f64-acf1-7836436b6c50\" (UID: \"39a45c2e-274a-4f64-acf1-7836436b6c50\") " Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258050 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-socket" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258433 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-socket" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258738 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-credential-socket" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258845 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258868 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258885 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258902 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a45c2e-274a-4f64-acf1-7836436b6c50-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.258917 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.259162 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-data" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.259358 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-certs" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.259529 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fe2fec-540b-473f-8303-46752a067bdc-istiod-ca-cert" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.259588 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-credential-socket" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.260004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.259815 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-credential-socket" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.261434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.260055 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-envoy" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.261434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.260162 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-data" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.261434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.260200 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb518472-d243-4a01-aead-146570bff41f-istiod-ca-cert" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:03.261434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.260249 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-certs" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.261434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.260550 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-socket" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.261434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.261405 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-kube-api-access-6x8xq" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "kube-api-access-6x8xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.262029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.261950 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9fe2fec-540b-473f-8303-46752a067bdc-istio-podinfo" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:03.262029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.261968 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-kube-api-access-45v4x" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "kube-api-access-45v4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.262029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.261999 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-istio-token" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.262029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.262070 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-token" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.262722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.262693 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-envoy" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.262722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.262710 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-envoy" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.263381 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.263362 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-istio-token" pod "e9fe2fec-540b-473f-8303-46752a067bdc" (UID: "e9fe2fec-540b-473f-8303-46752a067bdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.263657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.263631 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-podinfo" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:03.263972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.263950 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-kube-api-access-6w4t2" pod "39a45c2e-274a-4f64-acf1-7836436b6c50" (UID: "39a45c2e-274a-4f64-acf1-7836436b6c50"). InnerVolumeSpecName "kube-api-access-6w4t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.264203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.264186 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb518472-d243-4a01-aead-146570bff41f-istio-podinfo" pod "fb518472-d243-4a01-aead-146570bff41f" (UID: "fb518472-d243-4a01-aead-146570bff41f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:03.360162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360120 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-credential-socket\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360184 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-data\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360225 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb031617-7de9-4160-ad6b-762744f3c105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb031617-7de9-4160-ad6b-762744f3c105-istiod-ca-cert\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360260 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-istio-token\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-istio-token\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360308 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-kube-api-access-z66c5\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-kube-api-access-z66c5\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360353 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360350 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb031617-7de9-4160-ad6b-762744f3c105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb031617-7de9-4160-ad6b-762744f3c105-istio-podinfo\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360406 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-socket\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360440 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-certs\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.360600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360438 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-credential-socket" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.360600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360529 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-envoy\") pod \"eb031617-7de9-4160-ad6b-762744f3c105\" (UID: \"eb031617-7de9-4160-ad6b-762744f3c105\") " Nov 02 22:33:03.361017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360985 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-socket" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.361017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.360999 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-certs" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.361291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361269 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361297 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361313 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361327 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361340 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361354 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361367 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361382 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361396 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45v4x\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-kube-api-access-45v4x\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361409 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361422 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361435 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361449 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361462 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361474 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb518472-d243-4a01-aead-146570bff41f-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361489 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fe2fec-540b-473f-8303-46752a067bdc-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361504 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361517 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb518472-d243-4a01-aead-146570bff41f-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361535 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fe2fec-540b-473f-8303-46752a067bdc-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361547 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a45c2e-274a-4f64-acf1-7836436b6c50-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361561 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w4t2\" (UniqueName: \"kubernetes.io/projected/39a45c2e-274a-4f64-acf1-7836436b6c50-kube-api-access-6w4t2\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361609 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8xq\" (UniqueName: \"kubernetes.io/projected/fb518472-d243-4a01-aead-146570bff41f-kube-api-access-6x8xq\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361622 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fe2fec-540b-473f-8303-46752a067bdc-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361636 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb518472-d243-4a01-aead-146570bff41f-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361649 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fe2fec-540b-473f-8303-46752a067bdc-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361589 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb031617-7de9-4160-ad6b-762744f3c105-istiod-ca-cert" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:03.361894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.361844 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-data" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.363810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.363731 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-envoy" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:03.364176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.364149 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb031617-7de9-4160-ad6b-762744f3c105-istio-podinfo" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:03.365364 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.365295 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-kube-api-access-z66c5" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "kube-api-access-z66c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.365364 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.365321 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-istio-token" pod "eb031617-7de9-4160-ad6b-762744f3c105" (UID: "eb031617-7de9-4160-ad6b-762744f3c105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.463125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.463080 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb031617-7de9-4160-ad6b-762744f3c105-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.463125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.463120 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.463125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.463133 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb031617-7de9-4160-ad6b-762744f3c105-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.463461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.463147 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb031617-7de9-4160-ad6b-762744f3c105-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.463461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.463160 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.463461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.463178 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66c5\" (UniqueName: \"kubernetes.io/projected/eb031617-7de9-4160-ad6b-762744f3c105-kube-api-access-z66c5\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.646175 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.645738 5109 generic.go:356] "Generic (PLEG): container finished" podID="e9fe2fec-540b-473f-8303-46752a067bdc" containerID="6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b" exitCode=0 Nov 02 22:33:03.646175 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.646038 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" Nov 02 22:33:03.647340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.646033 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" event={"ID":"e9fe2fec-540b-473f-8303-46752a067bdc","Type":"ContainerDied","Data":"6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b"} Nov 02 22:33:03.647340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.647123 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75" event={"ID":"e9fe2fec-540b-473f-8303-46752a067bdc","Type":"ContainerDied","Data":"664d75c6cb75047a03bd3307ddbbe8a4df305d9569edd72b842791218b56e9c8"} Nov 02 22:33:03.647340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.647172 5109 scope.go:122] "RemoveContainer" containerID="6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b" Nov 02 22:33:03.653020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.650420 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" Nov 02 22:33:03.653020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.650420 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8" event={"ID":"c01b4628-5e7f-486a-9dd2-b5d47cf8df81","Type":"ContainerDied","Data":"e56fadbbb8528c9afda1933b0f083bad0e55e8b2331eca4d9870f074b8ed88c5"} Nov 02 22:33:03.653020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.650509 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e56fadbbb8528c9afda1933b0f083bad0e55e8b2331eca4d9870f074b8ed88c5" Nov 02 22:33:03.653907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.653877 5109 generic.go:356] "Generic (PLEG): container finished" podID="fb518472-d243-4a01-aead-146570bff41f" containerID="969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3" exitCode=0 Nov 02 22:33:03.654030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.653965 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" event={"ID":"fb518472-d243-4a01-aead-146570bff41f","Type":"ContainerDied","Data":"969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3"} Nov 02 22:33:03.654107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.653974 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" Nov 02 22:33:03.654107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.654057 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d" event={"ID":"fb518472-d243-4a01-aead-146570bff41f","Type":"ContainerDied","Data":"49e8fd79313fd2a49b7a7ddfe55bcf996bf5ddbe9a944d48702d74504ba9b981"} Nov 02 22:33:03.662698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.662662 5109 generic.go:356] "Generic (PLEG): container finished" podID="eb031617-7de9-4160-ad6b-762744f3c105" containerID="c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882" exitCode=0 Nov 02 22:33:03.662840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.662779 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" event={"ID":"eb031617-7de9-4160-ad6b-762744f3c105","Type":"ContainerDied","Data":"c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882"} Nov 02 22:33:03.662840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.662835 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" event={"ID":"eb031617-7de9-4160-ad6b-762744f3c105","Type":"ContainerDied","Data":"4c2b9d47b90716395030bd769c12233ccae8d341948ca1b6b43630a6df5a4aad"} Nov 02 22:33:03.662967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.662952 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6" Nov 02 22:33:03.663956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.663794 5109 scope.go:122] "RemoveContainer" containerID="6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b" Nov 02 22:33:03.664205 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:03.664152 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b\": container with ID starting with 6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b not found: ID does not exist" containerID="6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b" Nov 02 22:33:03.664325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.664191 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b"} err="failed to get container status \"6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b\": rpc error: code = NotFound desc = could not find container \"6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b\": container with ID starting with 6298bfc1784de65ebdf34f6cd3114513a1096dcdad04c7b6e7a39acde8374b8b not found: ID does not exist" Nov 02 22:33:03.664325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.664218 5109 scope.go:122] "RemoveContainer" containerID="969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3" Nov 02 22:33:03.666097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.665744 5109 generic.go:356] "Generic (PLEG): container finished" podID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerID="8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3" exitCode=0 Nov 02 22:33:03.666097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.665898 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" event={"ID":"39a45c2e-274a-4f64-acf1-7836436b6c50","Type":"ContainerDied","Data":"8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3"} Nov 02 22:33:03.666097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.666021 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" event={"ID":"39a45c2e-274a-4f64-acf1-7836436b6c50","Type":"ContainerDied","Data":"442bd0c6ea05106bbc994bcad2403cc1654d15fdb8e8a6deb696287ae3edc541"} Nov 02 22:33:03.667251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.667073 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln" Nov 02 22:33:03.691095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.689531 5109 scope.go:122] "RemoveContainer" containerID="969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3" Nov 02 22:33:03.691095 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:03.689939 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3\": container with ID starting with 969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3 not found: ID does not exist" containerID="969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3" Nov 02 22:33:03.691095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.689971 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3"} err="failed to get container status \"969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3\": rpc error: code = NotFound desc = could not find container \"969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3\": container with ID starting with 969702d3ea02b331009d52c7036e5dbeb24e9c22bf798cc0bc188978f10b0ce3 not found: ID does not exist" Nov 02 22:33:03.691095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.689990 5109 scope.go:122] "RemoveContainer" containerID="c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882" Nov 02 22:33:03.715989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.715865 5109 scope.go:122] "RemoveContainer" containerID="c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882" Nov 02 22:33:03.716762 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:03.716343 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882\": container with ID starting with c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882 not found: ID does not exist" containerID="c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882" Nov 02 22:33:03.716762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.716376 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882"} err="failed to get container status \"c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882\": rpc error: code = NotFound desc = could not find container \"c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882\": container with ID starting with c2ac63ec22e495e91ac7b84315c2dede6d32db12bdb894782717a45ddbb4c882 not found: ID does not exist" Nov 02 22:33:03.716762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.716397 5109 scope.go:122] "RemoveContainer" containerID="8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3" Nov 02 22:33:03.719784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.719629 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc"] Nov 02 22:33:03.724279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.724254 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98a222-77fdd496c9-vttjc"] Nov 02 22:33:03.731792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.731687 5109 scope.go:122] "RemoveContainer" containerID="8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3" Nov 02 22:33:03.732241 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:03.732189 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3\": container with ID starting with 8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3 not found: ID does not exist" containerID="8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3" Nov 02 22:33:03.732241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.732219 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3"} err="failed to get container status \"8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3\": rpc error: code = NotFound desc = could not find container \"8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3\": container with ID starting with 8d24f78eb45489b17c070176aaf7ec9d81e6ac7c8138026f77aa2fc20ac60cc3 not found: ID does not exist" Nov 02 22:33:03.864623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.860724 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:03.895624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.895590 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57460eb1-1c58-44dd-b7a8-b2f9ad12e627" path="/var/lib/kubelet/pods/57460eb1-1c58-44dd-b7a8-b2f9ad12e627/volumes" Nov 02 22:33:03.896095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.896076 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4b4ea2-9005-4fcf-9de2-29a19050ca73" path="/var/lib/kubelet/pods/5a4b4ea2-9005-4fcf-9de2-29a19050ca73/volumes" Nov 02 22:33:03.896549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.896499 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b23b36d-9139-46ea-98fe-5fd739865b32" path="/var/lib/kubelet/pods/8b23b36d-9139-46ea-98fe-5fd739865b32/volumes" Nov 02 22:33:03.903072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.903048 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:33:03.905150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.905130 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:33:03.970983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.970955 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-xks-ca\") pod \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " Nov 02 22:33:03.971162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971070 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-kube-api-access-28rl4\" (UniqueName: \"kubernetes.io/projected/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-kube-api-access-28rl4\") pod \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " Nov 02 22:33:03.971162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971113 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-openshift-ca\") pod \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " Nov 02 22:33:03.971162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971133 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-xks-ca\") pod \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\" (UID: \"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690\") " Nov 02 22:33:03.971344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971176 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-kube-api-access-h2ltc\" (UniqueName: \"kubernetes.io/projected/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-kube-api-access-h2ltc\") pod \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " Nov 02 22:33:03.971344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971194 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658ea0ef-0203-4ad7-982a-274f23cd6d37-kube-api-access-kspgs\" (UniqueName: \"kubernetes.io/projected/658ea0ef-0203-4ad7-982a-274f23cd6d37-kube-api-access-kspgs\") pod \"658ea0ef-0203-4ad7-982a-274f23cd6d37\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " Nov 02 22:33:03.971344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971275 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-xks-ca\") pod \"658ea0ef-0203-4ad7-982a-274f23cd6d37\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " Nov 02 22:33:03.971344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971318 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-openshift-ca\") pod \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\" (UID: \"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3\") " Nov 02 22:33:03.971555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971361 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-openshift-ca\") pod \"658ea0ef-0203-4ad7-982a-274f23cd6d37\" (UID: \"658ea0ef-0203-4ad7-982a-274f23cd6d37\") " Nov 02 22:33:03.971555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971502 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-openshift-ca" pod "1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" (UID: "1d2cb4bc-34ca-4fbb-918d-003b7b7d8690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:03.971976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.971948 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-openshift-ca" pod "f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" (UID: "f59aff8e-0f7e-4217-921a-ccb7a5e24ac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:03.972115 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.972010 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:03.972322 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.972294 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-openshift-ca" pod "658ea0ef-0203-4ad7-982a-274f23cd6d37" (UID: "658ea0ef-0203-4ad7-982a-274f23cd6d37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:03.974210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.974177 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658ea0ef-0203-4ad7-982a-274f23cd6d37-kube-api-access-kspgs" pod "658ea0ef-0203-4ad7-982a-274f23cd6d37" (UID: "658ea0ef-0203-4ad7-982a-274f23cd6d37"). InnerVolumeSpecName "kube-api-access-kspgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.974403 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.974374 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-kube-api-access-h2ltc" pod "f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" (UID: "f59aff8e-0f7e-4217-921a-ccb7a5e24ac3"). InnerVolumeSpecName "kube-api-access-h2ltc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:03.974898 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.974855 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-xks-ca" pod "f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" (UID: "f59aff8e-0f7e-4217-921a-ccb7a5e24ac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:03.975070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.975045 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-xks-ca" pod "1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" (UID: "1d2cb4bc-34ca-4fbb-918d-003b7b7d8690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:03.975469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.975443 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-xks-ca" pod "658ea0ef-0203-4ad7-982a-274f23cd6d37" (UID: "658ea0ef-0203-4ad7-982a-274f23cd6d37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:03.976390 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:03.976371 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-kube-api-access-28rl4" pod "1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" (UID: "1d2cb4bc-34ca-4fbb-918d-003b7b7d8690"). InnerVolumeSpecName "kube-api-access-28rl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:04.025253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.025213 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6"] Nov 02 22:33:04.028779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.028733 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-wn6v6"] Nov 02 22:33:04.085072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085040 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28rl4\" (UniqueName: \"kubernetes.io/projected/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-kube-api-access-28rl4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.085072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085070 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.085280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085081 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ltc\" (UniqueName: \"kubernetes.io/projected/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-kube-api-access-h2ltc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.085280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085091 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kspgs\" (UniqueName: \"kubernetes.io/projected/658ea0ef-0203-4ad7-982a-274f23cd6d37-kube-api-access-kspgs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.085280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085099 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.085280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085109 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.085280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085118 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658ea0ef-0203-4ad7-982a-274f23cd6d37-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.085280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.085126 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:04.144728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.144686 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75"] Nov 02 22:33:04.152024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.151940 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l6q75"] Nov 02 22:33:04.263346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.263313 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d"] Nov 02 22:33:04.267616 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.267554 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-xxn6d"] Nov 02 22:33:04.378075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.378046 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln"] Nov 02 22:33:04.382755 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.382725 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-dddln"] Nov 02 22:33:04.681127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.681099 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" Nov 02 22:33:04.681711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.681147 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc" event={"ID":"f59aff8e-0f7e-4217-921a-ccb7a5e24ac3","Type":"ContainerDied","Data":"fd63c41590ef833b5f31e145fc360106bcf444062d92efb2e40453c8bd07099b"} Nov 02 22:33:04.681805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.681748 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd63c41590ef833b5f31e145fc360106bcf444062d92efb2e40453c8bd07099b" Nov 02 22:33:04.684653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.684630 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" Nov 02 22:33:04.684797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.684657 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9" event={"ID":"1d2cb4bc-34ca-4fbb-918d-003b7b7d8690","Type":"ContainerDied","Data":"6ab33b7f8f5256e4b3e69a71afa7afe5270ec1fb1704b29af68cbdc1a450b58a"} Nov 02 22:33:04.684797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.684715 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ab33b7f8f5256e4b3e69a71afa7afe5270ec1fb1704b29af68cbdc1a450b58a" Nov 02 22:33:04.690924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.690903 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" Nov 02 22:33:04.691065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.690949 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4" event={"ID":"658ea0ef-0203-4ad7-982a-274f23cd6d37","Type":"ContainerDied","Data":"f384fcb0db6e69173314c28b65f0d1e9b83d155ee94d94b3ca50426875ccd971"} Nov 02 22:33:04.691065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.691005 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f384fcb0db6e69173314c28b65f0d1e9b83d155ee94d94b3ca50426875ccd971" Nov 02 22:33:04.840669 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.840627 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" podUID="500893a0-a2f3-4840-b9ee-59e5226d9335" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:04.840864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:04.840627 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" podUID="4da2ea58-8933-496e-a7d2-7711d131c55d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:05.747928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:05.747884 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" containerID="cri-o://7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f" gracePeriod=45 Nov 02 22:33:05.894705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:05.894665 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" path="/var/lib/kubelet/pods/39a45c2e-274a-4f64-acf1-7836436b6c50/volumes" Nov 02 22:33:05.895257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:05.895241 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" path="/var/lib/kubelet/pods/e9fe2fec-540b-473f-8303-46752a067bdc/volumes" Nov 02 22:33:05.895727 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:05.895713 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb031617-7de9-4160-ad6b-762744f3c105" path="/var/lib/kubelet/pods/eb031617-7de9-4160-ad6b-762744f3c105/volumes" Nov 02 22:33:05.896591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:05.896551 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb518472-d243-4a01-aead-146570bff41f" path="/var/lib/kubelet/pods/fb518472-d243-4a01-aead-146570bff41f/volumes" Nov 02 22:33:06.026500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.026472 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:33:06.108659 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.108612 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5155c18-83ec-4923-9855-dd0c6f293187-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5155c18-83ec-4923-9855-dd0c6f293187-tls-certs\") pod \"f5155c18-83ec-4923-9855-dd0c6f293187\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " Nov 02 22:33:06.108843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.108682 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5155c18-83ec-4923-9855-dd0c6f293187-kube-api-access-9k7zt\" (UniqueName: \"kubernetes.io/projected/f5155c18-83ec-4923-9855-dd0c6f293187-kube-api-access-9k7zt\") pod \"f5155c18-83ec-4923-9855-dd0c6f293187\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " Nov 02 22:33:06.108843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.108765 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-home\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-home\") pod \"f5155c18-83ec-4923-9855-dd0c6f293187\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " Nov 02 22:33:06.108843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.108829 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-tmp-dir\") pod \"f5155c18-83ec-4923-9855-dd0c6f293187\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " Nov 02 22:33:06.109038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.108884 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-model-cache\") pod \"f5155c18-83ec-4923-9855-dd0c6f293187\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " Nov 02 22:33:06.109038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.108946 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-dshm\") pod \"f5155c18-83ec-4923-9855-dd0c6f293187\" (UID: \"f5155c18-83ec-4923-9855-dd0c6f293187\") " Nov 02 22:33:06.109140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.109079 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-home" pod "f5155c18-83ec-4923-9855-dd0c6f293187" (UID: "f5155c18-83ec-4923-9855-dd0c6f293187"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.109140 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.109103 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-tmp-dir" pod "f5155c18-83ec-4923-9855-dd0c6f293187" (UID: "f5155c18-83ec-4923-9855-dd0c6f293187"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.109277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.109210 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-model-cache" pod "f5155c18-83ec-4923-9855-dd0c6f293187" (UID: "f5155c18-83ec-4923-9855-dd0c6f293187"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.109556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.109538 5109 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-home\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.109653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.109597 5109 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-tmp-dir\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.109653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.109615 5109 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-model-cache\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.111262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.111240 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5155c18-83ec-4923-9855-dd0c6f293187-kube-api-access-9k7zt" pod "f5155c18-83ec-4923-9855-dd0c6f293187" (UID: "f5155c18-83ec-4923-9855-dd0c6f293187"). InnerVolumeSpecName "kube-api-access-9k7zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:06.111354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.111337 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5155c18-83ec-4923-9855-dd0c6f293187-tls-certs" pod "f5155c18-83ec-4923-9855-dd0c6f293187" (UID: "f5155c18-83ec-4923-9855-dd0c6f293187"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:06.111510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.111497 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-dshm" pod "f5155c18-83ec-4923-9855-dd0c6f293187" (UID: "f5155c18-83ec-4923-9855-dd0c6f293187"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.210533 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.210494 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k7zt\" (UniqueName: \"kubernetes.io/projected/f5155c18-83ec-4923-9855-dd0c6f293187-kube-api-access-9k7zt\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.210533 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.210535 5109 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5155c18-83ec-4923-9855-dd0c6f293187-dshm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.210808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.210550 5109 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5155c18-83ec-4923-9855-dd0c6f293187-tls-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.713728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.713683 5109 generic.go:356] "Generic (PLEG): container finished" podID="f5155c18-83ec-4923-9855-dd0c6f293187" containerID="7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f" exitCode=0 Nov 02 22:33:06.713728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.713728 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" Nov 02 22:33:06.713993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.713730 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" event={"ID":"f5155c18-83ec-4923-9855-dd0c6f293187","Type":"ContainerDied","Data":"7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f"} Nov 02 22:33:06.713993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.713777 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72" event={"ID":"f5155c18-83ec-4923-9855-dd0c6f293187","Type":"ContainerDied","Data":"f0afed405481439d40d07127ea96793a28fc6f3dce4b8f791a04ec0f630fb974"} Nov 02 22:33:06.713993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.713795 5109 scope.go:122] "RemoveContainer" containerID="7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f" Nov 02 22:33:06.726143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.726117 5109 scope.go:122] "RemoveContainer" containerID="7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f" Nov 02 22:33:06.726496 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:06.726472 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f\": container with ID starting with 7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f not found: ID does not exist" containerID="7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f" Nov 02 22:33:06.726600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.726511 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f"} err="failed to get container status \"7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f\": rpc error: code = NotFound desc = could not find container \"7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f\": container with ID starting with 7bd4a999df27135e2ac750dbc512c5376967589c2cb457e5d760ed5f076eb58f not found: ID does not exist" Nov 02 22:33:06.880121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.880082 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72"] Nov 02 22:33:06.929406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:06.929373 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/premium-w3-98a222-kserve-84c8ccf58f-9ft72"] Nov 02 22:33:07.892501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:07.892465 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" path="/var/lib/kubelet/pods/f5155c18-83ec-4923-9855-dd0c6f293187/volumes" Nov 02 22:33:09.106412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.106291 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf"] Nov 02 22:33:09.107084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.107051 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" containerID="cri-o://077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259" gracePeriod=30 Nov 02 22:33:09.128201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.128163 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw"] Nov 02 22:33:09.128462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.128429 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" containerID="cri-o://8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680" gracePeriod=30 Nov 02 22:33:09.147073 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.147031 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7"] Nov 02 22:33:09.147270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.147247 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" containerID="cri-o://3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272" gracePeriod=30 Nov 02 22:33:09.153838 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.153810 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q"] Nov 02 22:33:09.154031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:09.154009 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" containerID="cri-o://873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673" gracePeriod=30 Nov 02 22:33:10.873597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:10.873541 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" podUID="353686ce-7828-478a-b4ad-9f27c3fdc577" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:10.874088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:10.873631 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" podUID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:12.437420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.437390 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:33:12.525287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.525257 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:33:12.567843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.567809 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500893a0-a2f3-4840-b9ee-59e5226d9335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500893a0-a2f3-4840-b9ee-59e5226d9335-plugins-config-volume\") pod \"500893a0-a2f3-4840-b9ee-59e5226d9335\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " Nov 02 22:33:12.568028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.567947 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500893a0-a2f3-4840-b9ee-59e5226d9335-kube-api-access-wmsfx\" (UniqueName: \"kubernetes.io/projected/500893a0-a2f3-4840-b9ee-59e5226d9335-kube-api-access-wmsfx\") pod \"500893a0-a2f3-4840-b9ee-59e5226d9335\" (UID: \"500893a0-a2f3-4840-b9ee-59e5226d9335\") " Nov 02 22:33:12.568288 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.568264 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500893a0-a2f3-4840-b9ee-59e5226d9335-plugins-config-volume" pod "500893a0-a2f3-4840-b9ee-59e5226d9335" (UID: "500893a0-a2f3-4840-b9ee-59e5226d9335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:12.568526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.568500 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500893a0-a2f3-4840-b9ee-59e5226d9335-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:12.570287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.570258 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500893a0-a2f3-4840-b9ee-59e5226d9335-kube-api-access-wmsfx" pod "500893a0-a2f3-4840-b9ee-59e5226d9335" (UID: "500893a0-a2f3-4840-b9ee-59e5226d9335"). InnerVolumeSpecName "kube-api-access-wmsfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:12.669535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.669418 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da2ea58-8933-496e-a7d2-7711d131c55d-kube-api-access-ts9km\" (UniqueName: \"kubernetes.io/projected/4da2ea58-8933-496e-a7d2-7711d131c55d-kube-api-access-ts9km\") pod \"4da2ea58-8933-496e-a7d2-7711d131c55d\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " Nov 02 22:33:12.669779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.669617 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4da2ea58-8933-496e-a7d2-7711d131c55d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da2ea58-8933-496e-a7d2-7711d131c55d-plugins-config-volume\") pod \"4da2ea58-8933-496e-a7d2-7711d131c55d\" (UID: \"4da2ea58-8933-496e-a7d2-7711d131c55d\") " Nov 02 22:33:12.670192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.670148 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsfx\" (UniqueName: \"kubernetes.io/projected/500893a0-a2f3-4840-b9ee-59e5226d9335-kube-api-access-wmsfx\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:12.670320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.670242 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da2ea58-8933-496e-a7d2-7711d131c55d-plugins-config-volume" pod "4da2ea58-8933-496e-a7d2-7711d131c55d" (UID: "4da2ea58-8933-496e-a7d2-7711d131c55d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:12.672408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.672383 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da2ea58-8933-496e-a7d2-7711d131c55d-kube-api-access-ts9km" pod "4da2ea58-8933-496e-a7d2-7711d131c55d" (UID: "4da2ea58-8933-496e-a7d2-7711d131c55d"). InnerVolumeSpecName "kube-api-access-ts9km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:12.767097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.767063 5109 generic.go:356] "Generic (PLEG): container finished" podID="4da2ea58-8933-496e-a7d2-7711d131c55d" containerID="37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5" exitCode=0 Nov 02 22:33:12.767281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.767110 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" Nov 02 22:33:12.767281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.767150 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" event={"ID":"4da2ea58-8933-496e-a7d2-7711d131c55d","Type":"ContainerDied","Data":"37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5"} Nov 02 22:33:12.767281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.767198 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g" event={"ID":"4da2ea58-8933-496e-a7d2-7711d131c55d","Type":"ContainerDied","Data":"dbfa6799032d45d90f406171347d721dd06b95db781dfe419924609cb163df68"} Nov 02 22:33:12.767281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.767222 5109 scope.go:122] "RemoveContainer" containerID="37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5" Nov 02 22:33:12.768999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.768753 5109 generic.go:356] "Generic (PLEG): container finished" podID="500893a0-a2f3-4840-b9ee-59e5226d9335" containerID="021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151" exitCode=0 Nov 02 22:33:12.768999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.768787 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" event={"ID":"500893a0-a2f3-4840-b9ee-59e5226d9335","Type":"ContainerDied","Data":"021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151"} Nov 02 22:33:12.768999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.768837 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" event={"ID":"500893a0-a2f3-4840-b9ee-59e5226d9335","Type":"ContainerDied","Data":"7ad27238fd6e25a66aa59f45bb9b5aebdc052122e4cfd70684139befe15a16b1"} Nov 02 22:33:12.768999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.768968 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c" Nov 02 22:33:12.771097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.771071 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts9km\" (UniqueName: \"kubernetes.io/projected/4da2ea58-8933-496e-a7d2-7711d131c55d-kube-api-access-ts9km\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:12.771189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.771100 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4da2ea58-8933-496e-a7d2-7711d131c55d-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:12.781220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.781202 5109 scope.go:122] "RemoveContainer" containerID="37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5" Nov 02 22:33:12.781487 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:12.781467 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5\": container with ID starting with 37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5 not found: ID does not exist" containerID="37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5" Nov 02 22:33:12.781529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.781496 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5"} err="failed to get container status \"37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5\": rpc error: code = NotFound desc = could not find container \"37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5\": container with ID starting with 37e912b07de37bbcdbdf8abc04e326af4edbe809d0e22c836bc98bc2c5fca5d5 not found: ID does not exist" Nov 02 22:33:12.781529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.781516 5109 scope.go:122] "RemoveContainer" containerID="021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151" Nov 02 22:33:12.792360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.792339 5109 scope.go:122] "RemoveContainer" containerID="021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151" Nov 02 22:33:12.792769 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:12.792748 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151\": container with ID starting with 021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151 not found: ID does not exist" containerID="021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151" Nov 02 22:33:12.792818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.792777 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151"} err="failed to get container status \"021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151\": rpc error: code = NotFound desc = could not find container \"021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151\": container with ID starting with 021cc22d2addf9c16ee12723afc831c9fc7d57fe464891e30f78d56a87bec151 not found: ID does not exist" Nov 02 22:33:12.935995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.935892 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c"] Nov 02 22:33:12.943684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:12.943654 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-49a557-6d4dcbff69-b4t5c"] Nov 02 22:33:13.055680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:13.055647 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g"] Nov 02 22:33:13.061548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:13.061517 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-49a557-7fb9fdcb59-wfx8g"] Nov 02 22:33:13.739460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:13.739421 5109 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:13.739865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:13.739482 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:13.892547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:13.892511 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da2ea58-8933-496e-a7d2-7711d131c55d" path="/var/lib/kubelet/pods/4da2ea58-8933-496e-a7d2-7711d131c55d/volumes" Nov 02 22:33:13.892966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:13.892950 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500893a0-a2f3-4840-b9ee-59e5226d9335" path="/var/lib/kubelet/pods/500893a0-a2f3-4840-b9ee-59e5226d9335/volumes" Nov 02 22:33:14.439458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.439431 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:33:14.496523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.496495 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:33:14.501787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.501761 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:33:14.523856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.523831 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:33:14.596347 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596317 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-kube-api-access-nvnc4\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-kube-api-access-nvnc4\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.596516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596356 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-certs\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.596516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596376 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-kube-api-access-qztmr\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-kube-api-access-qztmr\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.596516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596413 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-data\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.596516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596443 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-socket\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.596516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596480 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-data\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596530 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-credential-socket\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596627 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-podinfo\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596681 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-podinfo\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596710 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-istio-token\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-istio-token\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596710 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-data" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596757 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-envoy\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596759 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-certs" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596785 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-data\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.596834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596802 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-socket" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596823 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-podinfo\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596907 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-token\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-token\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596937 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-credential-socket\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.596983 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-kube-api-access-jftbc\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-kube-api-access-jftbc\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597017 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-envoy\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597035 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-credential-socket" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597084 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-certs\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.597294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597179 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/205f8467-9b38-454e-b15c-b97e553bee58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205f8467-9b38-454e-b15c-b97e553bee58-istio-podinfo\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597319 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b9c96c-a9f5-4212-aac8-2379505761d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9c96c-a9f5-4212-aac8-2379505761d7-istiod-ca-cert\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597347 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-envoy\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597380 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-token\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-token\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597409 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-data\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597445 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-credential-socket\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597477 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-kube-api-access-mlmb4\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-kube-api-access-mlmb4\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597539 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe162e44-0b4c-42a3-aacd-2abc635024b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe162e44-0b4c-42a3-aacd-2abc635024b8-istiod-ca-cert\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597621 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-envoy\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597674 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-certs\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.597757 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597721 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-socket\") pod \"fe162e44-0b4c-42a3-aacd-2abc635024b8\" (UID: \"fe162e44-0b4c-42a3-aacd-2abc635024b8\") " Nov 02 22:33:14.598467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597786 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-token\") pod \"a0b9c96c-a9f5-4212-aac8-2379505761d7\" (UID: \"a0b9c96c-a9f5-4212-aac8-2379505761d7\") " Nov 02 22:33:14.598467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597870 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-credential-socket\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.598467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597900 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-socket\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.598467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597960 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d2d9c2-5758-4033-a9b8-26e322a0af11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d2d9c2-5758-4033-a9b8-26e322a0af11-istiod-ca-cert\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.598467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597994 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-socket\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.598467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.598034 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205f8467-9b38-454e-b15c-b97e553bee58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205f8467-9b38-454e-b15c-b97e553bee58-istiod-ca-cert\") pod \"205f8467-9b38-454e-b15c-b97e553bee58\" (UID: \"205f8467-9b38-454e-b15c-b97e553bee58\") " Nov 02 22:33:14.598467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.598065 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-certs\") pod \"55d2d9c2-5758-4033-a9b8-26e322a0af11\" (UID: \"55d2d9c2-5758-4033-a9b8-26e322a0af11\") " Nov 02 22:33:14.598902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.598773 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.598902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.598795 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.598902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.598808 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.598902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.598821 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.603207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597233 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-data" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.597499 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-credential-socket" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603360 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205f8467-9b38-454e-b15c-b97e553bee58-istiod-ca-cert" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:14.603420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.599041 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-certs" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.599215 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-kube-api-access-nvnc4" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "kube-api-access-nvnc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.599457 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-podinfo" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:14.603420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.599983 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-istio-token" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.600045 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-kube-api-access-qztmr" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "kube-api-access-qztmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.600076 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-podinfo" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.600416 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-data" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.600525 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-token" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.600767 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-data" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.600804 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-credential-socket" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603446 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-envoy" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.600981 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b9c96c-a9f5-4212-aac8-2379505761d7-istiod-ca-cert" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.601000 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-envoy" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.601966 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-kube-api-access-mlmb4" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "kube-api-access-mlmb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603500 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-envoy" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.602008 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-kube-api-access-jftbc" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "kube-api-access-jftbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.602239 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-certs" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.602246 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-credential-socket" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.602421 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-socket" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.602487 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-token" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.602764 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d2d9c2-5758-4033-a9b8-26e322a0af11-istiod-ca-cert" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.602764 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-socket" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603028 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-socket" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603079 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe162e44-0b4c-42a3-aacd-2abc635024b8-istiod-ca-cert" pod "fe162e44-0b4c-42a3-aacd-2abc635024b8" (UID: "fe162e44-0b4c-42a3-aacd-2abc635024b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603142 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-token" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603343 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-certs" pod "a0b9c96c-a9f5-4212-aac8-2379505761d7" (UID: "a0b9c96c-a9f5-4212-aac8-2379505761d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603609 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-envoy" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.603780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.603661 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-podinfo" pod "55d2d9c2-5758-4033-a9b8-26e322a0af11" (UID: "55d2d9c2-5758-4033-a9b8-26e322a0af11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:14.604453 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.604265 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/205f8467-9b38-454e-b15c-b97e553bee58-istio-podinfo" pod "205f8467-9b38-454e-b15c-b97e553bee58" (UID: "205f8467-9b38-454e-b15c-b97e553bee58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700155 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700190 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700202 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700211 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700219 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700227 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700235 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700244 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700258 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jftbc\" (UniqueName: \"kubernetes.io/projected/55d2d9c2-5758-4033-a9b8-26e322a0af11-kube-api-access-jftbc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700273 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700282 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700290 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205f8467-9b38-454e-b15c-b97e553bee58-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700298 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9c96c-a9f5-4212-aac8-2379505761d7-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700306 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700314 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700322 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700331 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700345 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlmb4\" (UniqueName: \"kubernetes.io/projected/205f8467-9b38-454e-b15c-b97e553bee58-kube-api-access-mlmb4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700356 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe162e44-0b4c-42a3-aacd-2abc635024b8-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700366 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700374 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700383 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700391 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700400 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700408 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700422 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d2d9c2-5758-4033-a9b8-26e322a0af11-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700435 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205f8467-9b38-454e-b15c-b97e553bee58-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700443 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205f8467-9b38-454e-b15c-b97e553bee58-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700454 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d2d9c2-5758-4033-a9b8-26e322a0af11-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700462 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvnc4\" (UniqueName: \"kubernetes.io/projected/fe162e44-0b4c-42a3-aacd-2abc635024b8-kube-api-access-nvnc4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700471 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztmr\" (UniqueName: \"kubernetes.io/projected/a0b9c96c-a9f5-4212-aac8-2379505761d7-kube-api-access-qztmr\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.700621 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.700479 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe162e44-0b4c-42a3-aacd-2abc635024b8-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.791142 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.791101 5109 generic.go:356] "Generic (PLEG): container finished" podID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerID="8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680" exitCode=0 Nov 02 22:33:14.791629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.791170 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" Nov 02 22:33:14.791629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.791183 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" event={"ID":"fe162e44-0b4c-42a3-aacd-2abc635024b8","Type":"ContainerDied","Data":"8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680"} Nov 02 22:33:14.791629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.791264 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw" event={"ID":"fe162e44-0b4c-42a3-aacd-2abc635024b8","Type":"ContainerDied","Data":"1418da518a3f538bf605cf94054a6f901211dd024f91082d61cb4d13f5ff5967"} Nov 02 22:33:14.791629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.791295 5109 scope.go:122] "RemoveContainer" containerID="8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680" Nov 02 22:33:14.794382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.794245 5109 generic.go:356] "Generic (PLEG): container finished" podID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerID="873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673" exitCode=0 Nov 02 22:33:14.794382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.794281 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" event={"ID":"55d2d9c2-5758-4033-a9b8-26e322a0af11","Type":"ContainerDied","Data":"873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673"} Nov 02 22:33:14.794382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.794317 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" Nov 02 22:33:14.794382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.794341 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q" event={"ID":"55d2d9c2-5758-4033-a9b8-26e322a0af11","Type":"ContainerDied","Data":"7adcf0ff63e51af02503d3667b30619fa9fa5e473c02ca2c4058d98edbd0e02a"} Nov 02 22:33:14.796190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.796160 5109 generic.go:356] "Generic (PLEG): container finished" podID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerID="3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272" exitCode=0 Nov 02 22:33:14.796336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.796211 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" Nov 02 22:33:14.796336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.796230 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" event={"ID":"a0b9c96c-a9f5-4212-aac8-2379505761d7","Type":"ContainerDied","Data":"3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272"} Nov 02 22:33:14.796336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.796274 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7" event={"ID":"a0b9c96c-a9f5-4212-aac8-2379505761d7","Type":"ContainerDied","Data":"bd7f657d4d4edbeaaf4b1d5c753b8bf9d9d92c32b8239e0d27494b2b0280fde3"} Nov 02 22:33:14.799104 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.799076 5109 generic.go:356] "Generic (PLEG): container finished" podID="205f8467-9b38-454e-b15c-b97e553bee58" containerID="077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259" exitCode=0 Nov 02 22:33:14.799220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.799128 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" Nov 02 22:33:14.799220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.799145 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" event={"ID":"205f8467-9b38-454e-b15c-b97e553bee58","Type":"ContainerDied","Data":"077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259"} Nov 02 22:33:14.799220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.799182 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf" event={"ID":"205f8467-9b38-454e-b15c-b97e553bee58","Type":"ContainerDied","Data":"48f40c352ef73465ce30de4fa1847970648fd960c40362149fd2891cee8cdec1"} Nov 02 22:33:14.807796 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.807778 5109 scope.go:122] "RemoveContainer" containerID="8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680" Nov 02 22:33:14.808278 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:14.808249 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680\": container with ID starting with 8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680 not found: ID does not exist" containerID="8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680" Nov 02 22:33:14.808365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.808287 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680"} err="failed to get container status \"8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680\": rpc error: code = NotFound desc = could not find container \"8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680\": container with ID starting with 8e8c014e667cfe73a80f2d98ccba6b3e9571d12c32e24cef51945b1c709df680 not found: ID does not exist" Nov 02 22:33:14.808365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.808305 5109 scope.go:122] "RemoveContainer" containerID="873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673" Nov 02 22:33:14.819252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.819233 5109 scope.go:122] "RemoveContainer" containerID="877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a" Nov 02 22:33:14.829330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.829308 5109 scope.go:122] "RemoveContainer" containerID="873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673" Nov 02 22:33:14.829643 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:14.829622 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673\": container with ID starting with 873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673 not found: ID does not exist" containerID="873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673" Nov 02 22:33:14.829698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.829654 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673"} err="failed to get container status \"873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673\": rpc error: code = NotFound desc = could not find container \"873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673\": container with ID starting with 873b42ae42d7eb61742ce2f94e836d9fae3ca2a7ef1b8fe0b2945d446d431673 not found: ID does not exist" Nov 02 22:33:14.829698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.829674 5109 scope.go:122] "RemoveContainer" containerID="877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a" Nov 02 22:33:14.829955 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:14.829938 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a\": container with ID starting with 877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a not found: ID does not exist" containerID="877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a" Nov 02 22:33:14.829994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.829959 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a"} err="failed to get container status \"877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a\": rpc error: code = NotFound desc = could not find container \"877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a\": container with ID starting with 877dcdb34c8e14150eaa7791171d2a971aef794d2f487ff4f03843515484867a not found: ID does not exist" Nov 02 22:33:14.829994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.829971 5109 scope.go:122] "RemoveContainer" containerID="3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272" Nov 02 22:33:14.840438 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.840419 5109 scope.go:122] "RemoveContainer" containerID="3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272" Nov 02 22:33:14.840746 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:14.840722 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272\": container with ID starting with 3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272 not found: ID does not exist" containerID="3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272" Nov 02 22:33:14.840789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.840755 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272"} err="failed to get container status \"3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272\": rpc error: code = NotFound desc = could not find container \"3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272\": container with ID starting with 3a546e291698a017e7f66ec904b13d592387be9eaf2bc0692c08fe8744b9b272 not found: ID does not exist" Nov 02 22:33:14.840789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.840774 5109 scope.go:122] "RemoveContainer" containerID="077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259" Nov 02 22:33:14.852423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.852396 5109 scope.go:122] "RemoveContainer" containerID="077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259" Nov 02 22:33:14.852893 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:14.852865 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259\": container with ID starting with 077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259 not found: ID does not exist" containerID="077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259" Nov 02 22:33:14.853003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:14.852904 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259"} err="failed to get container status \"077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259\": rpc error: code = NotFound desc = could not find container \"077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259\": container with ID starting with 077f406bbbbcad17352af04bdb228a541da90ccb5b8c9fc2840bafdd1125b259 not found: ID does not exist" Nov 02 22:33:15.085020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.084978 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf"] Nov 02 22:33:15.089835 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.089807 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gkxzf"] Nov 02 22:33:15.203055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.202972 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw"] Nov 02 22:33:15.209537 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.209507 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-bwvnw"] Nov 02 22:33:15.319991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.319948 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q"] Nov 02 22:33:15.324108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.324074 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-fv95q"] Nov 02 22:33:15.436652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.436552 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7"] Nov 02 22:33:15.445374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.445345 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-4x5l7"] Nov 02 22:33:15.892290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.892257 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205f8467-9b38-454e-b15c-b97e553bee58" path="/var/lib/kubelet/pods/205f8467-9b38-454e-b15c-b97e553bee58/volumes" Nov 02 22:33:15.892931 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.892913 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" path="/var/lib/kubelet/pods/55d2d9c2-5758-4033-a9b8-26e322a0af11/volumes" Nov 02 22:33:15.893472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.893454 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" path="/var/lib/kubelet/pods/a0b9c96c-a9f5-4212-aac8-2379505761d7/volumes" Nov 02 22:33:15.894199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:15.894182 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" path="/var/lib/kubelet/pods/fe162e44-0b4c-42a3-aacd-2abc635024b8/volumes" Nov 02 22:33:16.198723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.198641 5109 scope.go:122] "RemoveContainer" containerID="08614d4c2c52acb29553e643e43821f05133822325ff7349e294ec6c1cb5a7a5" Nov 02 22:33:16.210285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.210264 5109 scope.go:122] "RemoveContainer" containerID="e23e81b27b4c202d829ee9ed9af8cec11d2b959496da2bdf1a1b9cd07c8acc83" Nov 02 22:33:16.220708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.220686 5109 scope.go:122] "RemoveContainer" containerID="a2e03c865a0477381c46f32212b24b50dae21b99834e7a461eec57419e0c1edc" Nov 02 22:33:16.230678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.230654 5109 scope.go:122] "RemoveContainer" containerID="08e508a60bc3da93fa1a7f639aca97daa78d4e3f79b9f015aee979ea36cf13cf" Nov 02 22:33:16.240724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.240698 5109 scope.go:122] "RemoveContainer" containerID="580ac40d65cdeac290a925c060466066141ae2c418fbfe97697ae44261c085f5" Nov 02 22:33:16.251538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.251516 5109 scope.go:122] "RemoveContainer" containerID="c77c668ec189f285fb298da8d61cd594ab4a859002c839d99cf82f5777de2cdb" Nov 02 22:33:16.261684 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.261664 5109 scope.go:122] "RemoveContainer" containerID="8378e2dcccd79f54842fbd0be7f4cce4d7615ec2554c9dd5db37589712585608" Nov 02 22:33:16.271283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.271258 5109 scope.go:122] "RemoveContainer" containerID="df4224617a59da99c388e365ea549aad27b1a520860f823f1ec405700e4040f4" Nov 02 22:33:16.285014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.284765 5109 scope.go:122] "RemoveContainer" containerID="f04b848cf5d66bd1a76bfbd49eefc259b8bc783b0dda73d7b3b71d101091b253" Nov 02 22:33:16.295935 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.295913 5109 scope.go:122] "RemoveContainer" containerID="4c8952d0767f59ba524fa690f67b2a87bda34c7cc2e181499466efcd63fdda7e" Nov 02 22:33:16.305756 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.305737 5109 scope.go:122] "RemoveContainer" containerID="6bd5ad4513c81311c606760b7c1c34432ebd87626ed3d2e8ecc9c25f5545d643" Nov 02 22:33:16.315615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.315596 5109 scope.go:122] "RemoveContainer" containerID="b31427295f0119854dbc58f1282807eae9229d21a279eb0f80c3a3113e312006" Nov 02 22:33:16.325977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.325958 5109 scope.go:122] "RemoveContainer" containerID="04afe1535124d4d1ab8758fba89e4cbe4b856cfea7be2fb89ac3d7c4090be4ae" Nov 02 22:33:16.335557 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.335536 5109 scope.go:122] "RemoveContainer" containerID="b58cea898bc3bdec047489d60c0b274c9002c2f601bda9a38bb53fcd347a87c4" Nov 02 22:33:16.345133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.345102 5109 scope.go:122] "RemoveContainer" containerID="5fd5d57c28e636dc0e04f62d3686a04a159996b121a6f0dc81aaa3f0d9ec3d2a" Nov 02 22:33:16.355019 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.354993 5109 scope.go:122] "RemoveContainer" containerID="bef548f21157debc5e9f51db88666475e0e717470133b04e33f733bff9bb6a5e" Nov 02 22:33:16.365049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:16.365027 5109 scope.go:122] "RemoveContainer" containerID="61238c1bb9b4660f514569a77f8f60dae061434a1366984e4563e0d3c2c854ad" Nov 02 22:33:18.635625 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:18.628135 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2"] Nov 02 22:33:18.683250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:18.682098 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-49a557-fbvc2"] Nov 02 22:33:19.999602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:19.970684 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5c1f3f-d551-449c-a605-4cda620cc9fe" path="/var/lib/kubelet/pods/bb5c1f3f-d551-449c-a605-4cda620cc9fe/volumes" Nov 02 22:33:20.031321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.029099 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95"] Nov 02 22:33:20.034462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.034094 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-98a222-pgw95"] Nov 02 22:33:20.038843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.038805 5109 generic.go:356] "Generic (PLEG): container finished" podID="353686ce-7828-478a-b4ad-9f27c3fdc577" containerID="b3b836ee829d9e345cdb67d4fe1918f37764e50313f4039608bf6ae147fb7a38" exitCode=0 Nov 02 22:33:20.039122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.039089 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" event={"ID":"353686ce-7828-478a-b4ad-9f27c3fdc577","Type":"ContainerDied","Data":"b3b836ee829d9e345cdb67d4fe1918f37764e50313f4039608bf6ae147fb7a38"} Nov 02 22:33:20.187973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.187938 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:33:20.223845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.223820 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:33:20.236807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.236770 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353686ce-7828-478a-b4ad-9f27c3fdc577-kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/353686ce-7828-478a-b4ad-9f27c3fdc577-kube-api-access-8jhbm\") pod \"353686ce-7828-478a-b4ad-9f27c3fdc577\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " Nov 02 22:33:20.236975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.236887 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353686ce-7828-478a-b4ad-9f27c3fdc577-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353686ce-7828-478a-b4ad-9f27c3fdc577-plugins-config-volume\") pod \"353686ce-7828-478a-b4ad-9f27c3fdc577\" (UID: \"353686ce-7828-478a-b4ad-9f27c3fdc577\") " Nov 02 22:33:20.236975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.236949 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-kube-api-access-qkw5n\" (UniqueName: \"kubernetes.io/projected/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-kube-api-access-qkw5n\") pod \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " Nov 02 22:33:20.237086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.237046 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-plugins-config-volume\") pod \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\" (UID: \"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385\") " Nov 02 22:33:20.237529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.237496 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353686ce-7828-478a-b4ad-9f27c3fdc577-plugins-config-volume" pod "353686ce-7828-478a-b4ad-9f27c3fdc577" (UID: "353686ce-7828-478a-b4ad-9f27c3fdc577"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:20.237850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.237525 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-plugins-config-volume" pod "4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" (UID: "4e4ea3c9-b253-455c-b55d-c1f9c2b0b385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:20.239254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.239232 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-kube-api-access-qkw5n" pod "4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" (UID: "4e4ea3c9-b253-455c-b55d-c1f9c2b0b385"). InnerVolumeSpecName "kube-api-access-qkw5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:20.239536 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.239507 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353686ce-7828-478a-b4ad-9f27c3fdc577-kube-api-access-8jhbm" pod "353686ce-7828-478a-b4ad-9f27c3fdc577" (UID: "353686ce-7828-478a-b4ad-9f27c3fdc577"). InnerVolumeSpecName "kube-api-access-8jhbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:20.338396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.338305 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:20.338396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.338348 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jhbm\" (UniqueName: \"kubernetes.io/projected/353686ce-7828-478a-b4ad-9f27c3fdc577-kube-api-access-8jhbm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:20.338396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.338363 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353686ce-7828-478a-b4ad-9f27c3fdc577-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:20.338396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:20.338378 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkw5n\" (UniqueName: \"kubernetes.io/projected/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385-kube-api-access-qkw5n\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:21.080612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.079146 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp"] Nov 02 22:33:21.080612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.079829 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b81f6b7b-5przp"] Nov 02 22:33:21.101630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.101217 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" Nov 02 22:33:21.101630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.101230 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8" event={"ID":"353686ce-7828-478a-b4ad-9f27c3fdc577","Type":"ContainerDied","Data":"a0fd26085e095cfdd56030b177d2d41b0566c024986c2d7fdaf465446889e13c"} Nov 02 22:33:21.101630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.101387 5109 scope.go:122] "RemoveContainer" containerID="b3b836ee829d9e345cdb67d4fe1918f37764e50313f4039608bf6ae147fb7a38" Nov 02 22:33:21.105679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.104964 5109 generic.go:356] "Generic (PLEG): container finished" podID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" containerID="613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029" exitCode=0 Nov 02 22:33:21.105679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.105035 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" Nov 02 22:33:21.105679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.105072 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" event={"ID":"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385","Type":"ContainerDied","Data":"613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029"} Nov 02 22:33:21.105679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.105135 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr" event={"ID":"4e4ea3c9-b253-455c-b55d-c1f9c2b0b385","Type":"ContainerDied","Data":"0a6a5e3cb22f7baf7c31c9a1e85e93c157b53abf9b9c0d4af7654bacb3269b9d"} Nov 02 22:33:21.116890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.116862 5109 scope.go:122] "RemoveContainer" containerID="613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029" Nov 02 22:33:21.131797 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.131770 5109 scope.go:122] "RemoveContainer" containerID="613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029" Nov 02 22:33:21.132232 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:21.132202 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029\": container with ID starting with 613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029 not found: ID does not exist" containerID="613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029" Nov 02 22:33:21.132351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.132245 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029"} err="failed to get container status \"613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029\": rpc error: code = NotFound desc = could not find container \"613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029\": container with ID starting with 613a5eff17394a347fcb9ba2d364417f84525e7465fdb53558467b3bc4d55029 not found: ID does not exist" Nov 02 22:33:21.193577 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.193525 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz"] Nov 02 22:33:21.197303 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.197272 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe810f-lbfqz"] Nov 02 22:33:21.272063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.272016 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8"] Nov 02 22:33:21.276058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.276025 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fe810f-f9c98bc45-5r8c8"] Nov 02 22:33:21.391237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.391130 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr"] Nov 02 22:33:21.394673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.394643 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe810f-774b7d8c4c-fgdhr"] Nov 02 22:33:21.892241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.892205 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01763f54-474e-48c8-b743-3c3e2dfdfca9" path="/var/lib/kubelet/pods/01763f54-474e-48c8-b743-3c3e2dfdfca9/volumes" Nov 02 22:33:21.892692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.892676 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323d2ab7-4eec-4a0c-98fb-18e795f6c77e" path="/var/lib/kubelet/pods/323d2ab7-4eec-4a0c-98fb-18e795f6c77e/volumes" Nov 02 22:33:21.893017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.893004 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353686ce-7828-478a-b4ad-9f27c3fdc577" path="/var/lib/kubelet/pods/353686ce-7828-478a-b4ad-9f27c3fdc577/volumes" Nov 02 22:33:21.893657 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.893643 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" path="/var/lib/kubelet/pods/4e4ea3c9-b253-455c-b55d-c1f9c2b0b385/volumes" Nov 02 22:33:21.894085 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:21.894071 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d823bfbc-36c0-486e-88b4-12e0b20e7ee3" path="/var/lib/kubelet/pods/d823bfbc-36c0-486e-88b4-12e0b20e7ee3/volumes" Nov 02 22:33:26.491426 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:26.491386 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:33:27.749290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:27.748612 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8"] Nov 02 22:33:27.749290 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:27.748880 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" containerID="cri-o://df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d" gracePeriod=30 Nov 02 22:33:28.757704 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.757665 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs"] Nov 02 22:33:28.758238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.757916 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" containerID="cri-o://f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e" gracePeriod=30 Nov 02 22:33:28.809833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.809787 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf"] Nov 02 22:33:28.810107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.810057 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" containerID="cri-o://b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0" gracePeriod=30 Nov 02 22:33:28.819234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.819198 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8"] Nov 02 22:33:28.819471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.819426 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" containerID="cri-o://fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda" gracePeriod=30 Nov 02 22:33:28.953478 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.953439 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t"] Nov 02 22:33:28.953781 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.953696 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" containerID="cri-o://0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1" gracePeriod=30 Nov 02 22:33:28.956943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.956913 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb"] Nov 02 22:33:28.957192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.957161 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" containerID="cri-o://68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c" gracePeriod=30 Nov 02 22:33:28.962044 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.962018 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r"] Nov 02 22:33:28.962247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:28.962223 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" containerID="cri-o://6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8" gracePeriod=30 Nov 02 22:33:29.697723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:29.691811 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv"] Nov 02 22:33:29.697723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:29.692502 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" containerID="cri-o://bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167" gracePeriod=30 Nov 02 22:33:32.306889 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:32.306854 5109 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:32.307520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:32.306913 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:33.018970 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.018945 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:33:33.061622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.061497 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-certs\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.061779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.061634 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-envoy\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.061779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.061667 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istiod-ca-cert\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.061779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.061710 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-podinfo\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.061779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.061745 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-credential-socket\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.062004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.061806 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-kube-api-access-9zhkp\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-kube-api-access-9zhkp\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.062004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.061873 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-socket\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.062130 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062037 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-data\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.062130 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062086 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istiod-ca-cert" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:33.062130 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062096 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-token\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-token\") pod \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\" (UID: \"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf\") " Nov 02 22:33:33.062299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062248 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-socket" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.062773 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062748 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.062987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062775 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.062987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062781 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-credential-socket" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.062987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062861 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-data" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.062987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.062866 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-certs" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.064405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.064357 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-podinfo" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:33.064405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.064351 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-envoy" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.064768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.064749 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-token" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:33.064843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.064780 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-kube-api-access-9zhkp" pod "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" (UID: "c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf"). InnerVolumeSpecName "kube-api-access-9zhkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:33.163784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.163747 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.163784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.163784 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.163784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.163794 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.164038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.163809 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.164038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.163818 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.164038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.163826 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.164038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.163838 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zhkp\" (UniqueName: \"kubernetes.io/projected/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf-kube-api-access-9zhkp\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.319772 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.319683 5109 generic.go:356] "Generic (PLEG): container finished" podID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerID="df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d" exitCode=0 Nov 02 22:33:33.319772 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.319730 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" Nov 02 22:33:33.319772 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.319753 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" event={"ID":"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf","Type":"ContainerDied","Data":"df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d"} Nov 02 22:33:33.320315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.319808 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8" event={"ID":"c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf","Type":"ContainerDied","Data":"d048289af1580b35f749f30111bd75cf44de1fcbfca712619fe9d57935ae0de5"} Nov 02 22:33:33.320315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.319832 5109 scope.go:122] "RemoveContainer" containerID="df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d" Nov 02 22:33:33.331529 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.331504 5109 scope.go:122] "RemoveContainer" containerID="df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d" Nov 02 22:33:33.331908 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:33.331883 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d\": container with ID starting with df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d not found: ID does not exist" containerID="df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d" Nov 02 22:33:33.331966 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.331920 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d"} err="failed to get container status \"df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d\": rpc error: code = NotFound desc = could not find container \"df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d\": container with ID starting with df8142df39aac9c9a7c4b023f3192d595544d0774f46c019024ab76e019c542d not found: ID does not exist" Nov 02 22:33:33.487900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.487863 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8"] Nov 02 22:33:33.494416 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.494381 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-hvcw8"] Nov 02 22:33:33.891956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:33.891901 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" path="/var/lib/kubelet/pods/c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf/volumes" Nov 02 22:33:34.180051 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.180024 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:33:34.187602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.186780 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:33:34.197168 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.195812 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:33:34.259107 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.259074 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277713 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-data\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277776 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-envoy\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277819 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-envoy\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277859 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-credential-socket\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277902 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-token\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-token\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277931 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-socket\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277961 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-kube-api-access-x8c4h\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-kube-api-access-x8c4h\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.277990 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-data\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278024 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-token\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-token\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278053 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-data\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278093 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c12691b-81b3-4967-837c-f32390a02cbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c12691b-81b3-4967-837c-f32390a02cbb-istiod-ca-cert\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278125 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c12691b-81b3-4967-837c-f32390a02cbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c12691b-81b3-4967-837c-f32390a02cbb-istio-podinfo\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278175 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-envoy\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278214 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-podinfo\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278265 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-kube-api-access-pgq5q\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-kube-api-access-pgq5q\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278330 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-certs\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278359 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-kube-api-access-cdqm7\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-kube-api-access-cdqm7\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278390 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-socket\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278413 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-certs\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278464 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242212e7-a571-4d7d-b880-c6ee0908cdf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242212e7-a571-4d7d-b880-c6ee0908cdf3-istiod-ca-cert\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278498 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-istio-token\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-istio-token\") pod \"5c12691b-81b3-4967-837c-f32390a02cbb\" (UID: \"5c12691b-81b3-4967-837c-f32390a02cbb\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278534 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-podinfo\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278605 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-credential-socket\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278626 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-credential-socket\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278689 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-socket\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278706 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1aa813-c5de-4562-91f7-ea2c64086c30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1aa813-c5de-4562-91f7-ea2c64086c30-istiod-ca-cert\") pod \"3b1aa813-c5de-4562-91f7-ea2c64086c30\" (UID: \"3b1aa813-c5de-4562-91f7-ea2c64086c30\") " Nov 02 22:33:34.278751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278741 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-certs\") pod \"242212e7-a571-4d7d-b880-c6ee0908cdf3\" (UID: \"242212e7-a571-4d7d-b880-c6ee0908cdf3\") " Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278021 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-data" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278301 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-credential-socket" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.278519 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-socket" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.279242 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-data" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.279290 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242212e7-a571-4d7d-b880-c6ee0908cdf3-istiod-ca-cert" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.279551 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-data" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.280262 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-certs" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.280930 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-envoy" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.282516 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-token" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.282837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.282821 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-certs" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.283323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283007 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-socket" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.283323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283133 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-istio-token" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.283323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283152 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-certs" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.283323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283310 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-envoy" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.283481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283324 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-credential-socket" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.283533 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283484 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-credential-socket" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.283533 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283493 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-socket" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.283822 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283761 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c12691b-81b3-4967-837c-f32390a02cbb-istiod-ca-cert" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.283822 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.283779 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1aa813-c5de-4562-91f7-ea2c64086c30-istiod-ca-cert" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.284520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.284446 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-podinfo" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:34.284520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.284479 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-kube-api-access-x8c4h" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "kube-api-access-x8c4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.284947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.284919 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-kube-api-access-cdqm7" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "kube-api-access-cdqm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.286546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.286519 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c12691b-81b3-4967-837c-f32390a02cbb-istio-podinfo" pod "5c12691b-81b3-4967-837c-f32390a02cbb" (UID: "5c12691b-81b3-4967-837c-f32390a02cbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:34.288087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.288065 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-envoy" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.288087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.288074 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-podinfo" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:34.288210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.288084 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-token" pod "3b1aa813-c5de-4562-91f7-ea2c64086c30" (UID: "3b1aa813-c5de-4562-91f7-ea2c64086c30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.288251 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.288208 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-kube-api-access-pgq5q" pod "242212e7-a571-4d7d-b880-c6ee0908cdf3" (UID: "242212e7-a571-4d7d-b880-c6ee0908cdf3"). InnerVolumeSpecName "kube-api-access-pgq5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.319059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.319029 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:33:34.324253 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.324211 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:33:34.331484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.331455 5109 generic.go:356] "Generic (PLEG): container finished" podID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerID="fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda" exitCode=0 Nov 02 22:33:34.331809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.331532 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" Nov 02 22:33:34.331809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.331592 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" event={"ID":"242212e7-a571-4d7d-b880-c6ee0908cdf3","Type":"ContainerDied","Data":"fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda"} Nov 02 22:33:34.331809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.331637 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8" event={"ID":"242212e7-a571-4d7d-b880-c6ee0908cdf3","Type":"ContainerDied","Data":"2f629b1412226adf6e06d8e5b4eabeab2d226546c3cb55af073983da3f5a063d"} Nov 02 22:33:34.331809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.331659 5109 scope.go:122] "RemoveContainer" containerID="fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda" Nov 02 22:33:34.333768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.333735 5109 generic.go:356] "Generic (PLEG): container finished" podID="5c12691b-81b3-4967-837c-f32390a02cbb" containerID="b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0" exitCode=0 Nov 02 22:33:34.333900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.333790 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" Nov 02 22:33:34.333900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.333818 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" event={"ID":"5c12691b-81b3-4967-837c-f32390a02cbb","Type":"ContainerDied","Data":"b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0"} Nov 02 22:33:34.333900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.333885 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf" event={"ID":"5c12691b-81b3-4967-837c-f32390a02cbb","Type":"ContainerDied","Data":"d8350510176c6be32941651a146c6876374001d9dd9a06f439560a3b1b9b9ae6"} Nov 02 22:33:34.335860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.335821 5109 generic.go:356] "Generic (PLEG): container finished" podID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerID="0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1" exitCode=0 Nov 02 22:33:34.336038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.336017 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" event={"ID":"40ce603b-0b3f-49f8-b801-3ba1d6c952e4","Type":"ContainerDied","Data":"0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1"} Nov 02 22:33:34.336242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.336072 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" event={"ID":"40ce603b-0b3f-49f8-b801-3ba1d6c952e4","Type":"ContainerDied","Data":"e2e113aa6e47a55f18fd14a9323dcf5f7922e5f0d18c71abe327f22e47ad2f95"} Nov 02 22:33:34.336242 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.336183 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t" Nov 02 22:33:34.339203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.339177 5109 generic.go:356] "Generic (PLEG): container finished" podID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerID="68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c" exitCode=0 Nov 02 22:33:34.339317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.339266 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" event={"ID":"50358e22-7a8b-456d-ab7b-41ee21f7c263","Type":"ContainerDied","Data":"68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c"} Nov 02 22:33:34.339317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.339283 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" Nov 02 22:33:34.339317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.339312 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb" event={"ID":"50358e22-7a8b-456d-ab7b-41ee21f7c263","Type":"ContainerDied","Data":"a494ff68966215800105d29761ca9ecda225f06280f83dee6961efb30b59c9fc"} Nov 02 22:33:34.340969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.340944 5109 generic.go:356] "Generic (PLEG): container finished" podID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerID="f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e" exitCode=0 Nov 02 22:33:34.341094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.340991 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" Nov 02 22:33:34.341094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.341030 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" event={"ID":"3b1aa813-c5de-4562-91f7-ea2c64086c30","Type":"ContainerDied","Data":"f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e"} Nov 02 22:33:34.341094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.341060 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs" event={"ID":"3b1aa813-c5de-4562-91f7-ea2c64086c30","Type":"ContainerDied","Data":"b13a2e149e277483f0607ede8c00750d4bb2f8cd5f2116260e7135967ba58f18"} Nov 02 22:33:34.343199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.343178 5109 generic.go:356] "Generic (PLEG): container finished" podID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerID="6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8" exitCode=0 Nov 02 22:33:34.343387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.343325 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" event={"ID":"db8f588d-d144-48c6-8fba-6e6f3a033343","Type":"ContainerDied","Data":"6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8"} Nov 02 22:33:34.343543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.343522 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" Nov 02 22:33:34.343696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.343522 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r" event={"ID":"db8f588d-d144-48c6-8fba-6e6f3a033343","Type":"ContainerDied","Data":"b6b032c59f75828920f7fc782f47679b8719934084e94cb328c616513788a063"} Nov 02 22:33:34.344670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.344651 5109 scope.go:122] "RemoveContainer" containerID="fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda" Nov 02 22:33:34.344957 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:34.344934 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda\": container with ID starting with fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda not found: ID does not exist" containerID="fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda" Nov 02 22:33:34.345066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.344970 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda"} err="failed to get container status \"fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda\": rpc error: code = NotFound desc = could not find container \"fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda\": container with ID starting with fd6f152b17e202a8c357bc37a8473fc05e0bf89e83ea9819c7599512ee050fda not found: ID does not exist" Nov 02 22:33:34.345066 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.344992 5109 scope.go:122] "RemoveContainer" containerID="b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0" Nov 02 22:33:34.355241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.355217 5109 scope.go:122] "RemoveContainer" containerID="b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0" Nov 02 22:33:34.355543 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:34.355524 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0\": container with ID starting with b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0 not found: ID does not exist" containerID="b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0" Nov 02 22:33:34.355633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.355550 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0"} err="failed to get container status \"b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0\": rpc error: code = NotFound desc = could not find container \"b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0\": container with ID starting with b35b6b86b610dd9891bd13ba50cfacd203b7b64a3843142060f17e03319adda0 not found: ID does not exist" Nov 02 22:33:34.355633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.355597 5109 scope.go:122] "RemoveContainer" containerID="0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1" Nov 02 22:33:34.365383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.365362 5109 scope.go:122] "RemoveContainer" containerID="0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1" Nov 02 22:33:34.365753 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:34.365728 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1\": container with ID starting with 0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1 not found: ID does not exist" containerID="0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1" Nov 02 22:33:34.365808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.365764 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1"} err="failed to get container status \"0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1\": rpc error: code = NotFound desc = could not find container \"0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1\": container with ID starting with 0bb46cfb5cd89eae5868ebe6ca3086c983d49ad6fa640906b95cd90a95807dc1 not found: ID does not exist" Nov 02 22:33:34.365808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.365783 5109 scope.go:122] "RemoveContainer" containerID="68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c" Nov 02 22:33:34.375648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.375625 5109 scope.go:122] "RemoveContainer" containerID="68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c" Nov 02 22:33:34.375981 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:34.375957 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c\": container with ID starting with 68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c not found: ID does not exist" containerID="68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c" Nov 02 22:33:34.376061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.375992 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c"} err="failed to get container status \"68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c\": rpc error: code = NotFound desc = could not find container \"68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c\": container with ID starting with 68cee1b0a6495bf64dcb38691e1fcb4dfa4919c3bc6926eeb571f6a6e94c394c not found: ID does not exist" Nov 02 22:33:34.376061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.376011 5109 scope.go:122] "RemoveContainer" containerID="f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e" Nov 02 22:33:34.381332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381304 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-token\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-token\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.381447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381353 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-kube-api-access-wqlbn\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-kube-api-access-wqlbn\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.381447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381407 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-certs\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.381447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381436 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istiod-ca-cert\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.381673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381462 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-credential-socket\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.381673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381507 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-credential-socket\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.381673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381545 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-socket\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.381673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381611 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8f588d-d144-48c6-8fba-6e6f3a033343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8f588d-d144-48c6-8fba-6e6f3a033343-istiod-ca-cert\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.381673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381642 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-podinfo\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381691 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-data\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381719 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-podinfo\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381745 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50358e22-7a8b-456d-ab7b-41ee21f7c263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50358e22-7a8b-456d-ab7b-41ee21f7c263-istiod-ca-cert\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381788 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-socket\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381817 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-kube-api-access-qkgzc\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-kube-api-access-qkgzc\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381846 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-credential-socket" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381875 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-socket\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381883 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istiod-ca-cert" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381910 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-data\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.381948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381943 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-podinfo\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.381975 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-credential-socket\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382023 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-certs\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382049 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-kube-api-access-flf99\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-kube-api-access-flf99\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382102 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-certs\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382116 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-socket" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382127 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-token\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-token\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382158 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-data\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382161 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-credential-socket" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382201 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-envoy\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382242 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-token\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-token\") pod \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\" (UID: \"40ce603b-0b3f-49f8-b801-3ba1d6c952e4\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382272 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-envoy\") pod \"50358e22-7a8b-456d-ab7b-41ee21f7c263\" (UID: \"50358e22-7a8b-456d-ab7b-41ee21f7c263\") " Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382286 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8f588d-d144-48c6-8fba-6e6f3a033343-istiod-ca-cert" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382592 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-data" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382335 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-certs" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382491 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-data" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382557 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-socket" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.382682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.382503 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-envoy\") pod \"db8f588d-d144-48c6-8fba-6e6f3a033343\" (UID: \"db8f588d-d144-48c6-8fba-6e6f3a033343\") " Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383164 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383215 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383278 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383351 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383452 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383467 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383484 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383500 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383515 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8c4h\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-kube-api-access-x8c4h\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383532 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383548 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8f588d-d144-48c6-8fba-6e6f3a033343-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383560 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.383558 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383590 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383606 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c12691b-81b3-4967-837c-f32390a02cbb-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383621 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c12691b-81b3-4967-837c-f32390a02cbb-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383640 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383657 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383672 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1aa813-c5de-4562-91f7-ea2c64086c30-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383687 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383737 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50358e22-7a8b-456d-ab7b-41ee21f7c263-istiod-ca-cert" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383749 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgq5q\" (UniqueName: \"kubernetes.io/projected/242212e7-a571-4d7d-b880-c6ee0908cdf3-kube-api-access-pgq5q\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383767 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383783 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdqm7\" (UniqueName: \"kubernetes.io/projected/3b1aa813-c5de-4562-91f7-ea2c64086c30-kube-api-access-cdqm7\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383801 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383816 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383830 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383845 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242212e7-a571-4d7d-b880-c6ee0908cdf3-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383860 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c12691b-81b3-4967-837c-f32390a02cbb-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383877 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383892 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242212e7-a571-4d7d-b880-c6ee0908cdf3-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383912 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383928 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383945 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa813-c5de-4562-91f7-ea2c64086c30-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383961 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1aa813-c5de-4562-91f7-ea2c64086c30-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383977 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242212e7-a571-4d7d-b880-c6ee0908cdf3-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.383992 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.384008 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c12691b-81b3-4967-837c-f32390a02cbb-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.384230 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-certs" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.384354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.384250 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-socket" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.385713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.384438 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-credential-socket" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.385713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.384948 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-data" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.385713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.385093 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-token" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.385713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.385378 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-certs" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.385943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.385810 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-kube-api-access-wqlbn" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "kube-api-access-wqlbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.386918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.386866 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-kube-api-access-qkgzc" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "kube-api-access-qkgzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.386918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.386890 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-envoy" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.387379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.387335 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-token" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.387628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.387605 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-envoy" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.387918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.387884 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-token" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.388006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.387912 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-podinfo" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:34.388201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.388182 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-podinfo" pod "db8f588d-d144-48c6-8fba-6e6f3a033343" (UID: "db8f588d-d144-48c6-8fba-6e6f3a033343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:34.388709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.388623 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-envoy" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:34.388709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.388661 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-podinfo" pod "40ce603b-0b3f-49f8-b801-3ba1d6c952e4" (UID: "40ce603b-0b3f-49f8-b801-3ba1d6c952e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:34.388799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.388765 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-kube-api-access-flf99" pod "50358e22-7a8b-456d-ab7b-41ee21f7c263" (UID: "50358e22-7a8b-456d-ab7b-41ee21f7c263"). InnerVolumeSpecName "kube-api-access-flf99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.388799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.388792 5109 scope.go:122] "RemoveContainer" containerID="f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e" Nov 02 22:33:34.389170 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:34.389147 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e\": container with ID starting with f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e not found: ID does not exist" containerID="f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e" Nov 02 22:33:34.389265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.389175 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e"} err="failed to get container status \"f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e\": rpc error: code = NotFound desc = could not find container \"f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e\": container with ID starting with f19514011922d405db7e835dd113fd82c39da8e9db9783e09eef3c8a317d9f2e not found: ID does not exist" Nov 02 22:33:34.389265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.389194 5109 scope.go:122] "RemoveContainer" containerID="6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8" Nov 02 22:33:34.399152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.399131 5109 scope.go:122] "RemoveContainer" containerID="6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8" Nov 02 22:33:34.399507 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:34.399483 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8\": container with ID starting with 6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8 not found: ID does not exist" containerID="6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8" Nov 02 22:33:34.399600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.399516 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8"} err="failed to get container status \"6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8\": rpc error: code = NotFound desc = could not find container \"6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8\": container with ID starting with 6fdd0b553045b052848691c4f02acf7d5fa0ba2894450414437ab012b9fe6ca8 not found: ID does not exist" Nov 02 22:33:34.484819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484785 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.484819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484816 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.484819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484825 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484834 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flf99\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-kube-api-access-flf99\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484846 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484854 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484862 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484870 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484878 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484886 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484894 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484902 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484909 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlbn\" (UniqueName: \"kubernetes.io/projected/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-kube-api-access-wqlbn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484917 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50358e22-7a8b-456d-ab7b-41ee21f7c263-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484925 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50358e22-7a8b-456d-ab7b-41ee21f7c263-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484932 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ce603b-0b3f-49f8-b801-3ba1d6c952e4-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484941 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50358e22-7a8b-456d-ab7b-41ee21f7c263-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.485169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.484949 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkgzc\" (UniqueName: \"kubernetes.io/projected/db8f588d-d144-48c6-8fba-6e6f3a033343-kube-api-access-qkgzc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.777167 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.777119 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r"] Nov 02 22:33:34.783874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.783839 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-h8f2r"] Nov 02 22:33:34.965877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.965779 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t"] Nov 02 22:33:34.970984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:34.970954 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-9mx4t"] Nov 02 22:33:35.081665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.081632 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf"] Nov 02 22:33:35.085520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.085486 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-wzfxf"] Nov 02 22:33:35.198120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.198085 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs"] Nov 02 22:33:35.200879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.200851 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-6j4zs"] Nov 02 22:33:35.378789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.378749 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb"] Nov 02 22:33:35.383074 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.383040 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6mrkb"] Nov 02 22:33:35.492598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.492532 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8"] Nov 02 22:33:35.496965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.496931 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4fbb8"] Nov 02 22:33:35.762337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.762299 5109 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:33:35.762508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.762361 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 22:33:35.864097 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.864071 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:33:35.893382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.893346 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" path="/var/lib/kubelet/pods/242212e7-a571-4d7d-b880-c6ee0908cdf3/volumes" Nov 02 22:33:35.893916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.893896 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" path="/var/lib/kubelet/pods/3b1aa813-c5de-4562-91f7-ea2c64086c30/volumes" Nov 02 22:33:35.894293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.894279 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" path="/var/lib/kubelet/pods/40ce603b-0b3f-49f8-b801-3ba1d6c952e4/volumes" Nov 02 22:33:35.895131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.895110 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" path="/var/lib/kubelet/pods/50358e22-7a8b-456d-ab7b-41ee21f7c263/volumes" Nov 02 22:33:35.895519 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.895506 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" path="/var/lib/kubelet/pods/5c12691b-81b3-4967-837c-f32390a02cbb/volumes" Nov 02 22:33:35.895913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.895900 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" path="/var/lib/kubelet/pods/db8f588d-d144-48c6-8fba-6e6f3a033343/volumes" Nov 02 22:33:36.000024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.999914 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-kube-api-access-mfpnh\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-kube-api-access-mfpnh\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:35.999972 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-envoy\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.000025 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-credential-socket\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.000111 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-data\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.000145 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-token\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.000185 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-podinfo\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.000234 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ec1893-7330-4af0-91d1-448ba3809d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ec1893-7330-4af0-91d1-448ba3809d8c-istiod-ca-cert\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000304 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.000261 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-certs\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.000683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.000321 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-socket\") pod \"23ec1893-7330-4af0-91d1-448ba3809d8c\" (UID: \"23ec1893-7330-4af0-91d1-448ba3809d8c\") " Nov 02 22:33:36.001255 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.001221 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-socket" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.001372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.001278 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ec1893-7330-4af0-91d1-448ba3809d8c-istiod-ca-cert" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:36.001434 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.001364 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-certs" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.002518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.002491 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-data" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.002670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.002539 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-credential-socket" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.002958 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.002922 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-envoy" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.004030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.004007 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-token" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:36.004176 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.004106 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-podinfo" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:36.004348 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.004323 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-kube-api-access-mfpnh" pod "23ec1893-7330-4af0-91d1-448ba3809d8c" (UID: "23ec1893-7330-4af0-91d1-448ba3809d8c"). InnerVolumeSpecName "kube-api-access-mfpnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:36.101383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101331 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101374 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfpnh\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-kube-api-access-mfpnh\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101389 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101401 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101413 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101424 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101436 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ec1893-7330-4af0-91d1-448ba3809d8c-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101446 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ec1893-7330-4af0-91d1-448ba3809d8c-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.101688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.101458 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ec1893-7330-4af0-91d1-448ba3809d8c-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.366031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.365992 5109 generic.go:356] "Generic (PLEG): container finished" podID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerID="bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167" exitCode=0 Nov 02 22:33:36.366285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.366070 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" Nov 02 22:33:36.366285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.366081 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" event={"ID":"23ec1893-7330-4af0-91d1-448ba3809d8c","Type":"ContainerDied","Data":"bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167"} Nov 02 22:33:36.366285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.366170 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv" event={"ID":"23ec1893-7330-4af0-91d1-448ba3809d8c","Type":"ContainerDied","Data":"fa3b98fc1e68550b3a0fd1df3a21585fc7d9230880545df1d27dcc2bc364415a"} Nov 02 22:33:36.366285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.366195 5109 scope.go:122] "RemoveContainer" containerID="bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167" Nov 02 22:33:36.377438 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.377419 5109 scope.go:122] "RemoveContainer" containerID="89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f" Nov 02 22:33:36.387597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.387557 5109 scope.go:122] "RemoveContainer" containerID="bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167" Nov 02 22:33:36.387902 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:36.387872 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167\": container with ID starting with bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167 not found: ID does not exist" containerID="bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167" Nov 02 22:33:36.387954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.387901 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167"} err="failed to get container status \"bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167\": rpc error: code = NotFound desc = could not find container \"bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167\": container with ID starting with bf5efa7ef85335ebb85d49b1aebf8db53d1e902215a3307da5c33dcd1f083167 not found: ID does not exist" Nov 02 22:33:36.387954 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.387921 5109 scope.go:122] "RemoveContainer" containerID="89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f" Nov 02 22:33:36.388200 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:36.388180 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f\": container with ID starting with 89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f not found: ID does not exist" containerID="89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f" Nov 02 22:33:36.388263 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.388209 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f"} err="failed to get container status \"89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f\": rpc error: code = NotFound desc = could not find container \"89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f\": container with ID starting with 89ce28e8f5a31d6c61c8ac9fe623037b4d26b2b703f427ba70d56943b272c54f not found: ID does not exist" Nov 02 22:33:36.533001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.532966 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv"] Nov 02 22:33:36.537321 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:36.537292 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-hw4rv"] Nov 02 22:33:37.891247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:37.891213 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" path="/var/lib/kubelet/pods/23ec1893-7330-4af0-91d1-448ba3809d8c/volumes" Nov 02 22:33:51.806345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806309 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv"] Nov 02 22:33:51.806789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806765 5109 cpu_manager.go:414] "containerMap: removing container" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" Nov 02 22:33:51.806789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806778 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806792 5109 cpu_manager.go:414] "containerMap: removing container" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806797 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806804 5109 cpu_manager.go:414] "containerMap: removing container" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806809 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806818 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806822 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806831 5109 cpu_manager.go:414] "containerMap: removing container" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806835 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806844 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" containerName="revoke-keys" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806848 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" containerName="revoke-keys" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806857 5109 cpu_manager.go:414] "containerMap: removing container" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806861 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" Nov 02 22:33:51.806864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806867 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806871 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806879 5109 cpu_manager.go:414] "containerMap: removing container" podUID="4da2ea58-8933-496e-a7d2-7711d131c55d" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806884 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da2ea58-8933-496e-a7d2-7711d131c55d" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806891 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c01b4628-5e7f-486a-9dd2-b5d47cf8df81" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806896 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01b4628-5e7f-486a-9dd2-b5d47cf8df81" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806904 5109 cpu_manager.go:414] "containerMap: removing container" podUID="353686ce-7828-478a-b4ad-9f27c3fdc577" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806908 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353686ce-7828-478a-b4ad-9f27c3fdc577" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806915 5109 cpu_manager.go:414] "containerMap: removing container" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806919 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806925 5109 cpu_manager.go:414] "containerMap: removing container" podUID="57460eb1-1c58-44dd-b7a8-b2f9ad12e627" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806930 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57460eb1-1c58-44dd-b7a8-b2f9ad12e627" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806939 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806943 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806950 5109 cpu_manager.go:414] "containerMap: removing container" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806955 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806963 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806967 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806972 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806977 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806983 5109 cpu_manager.go:414] "containerMap: removing container" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806987 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806995 5109 cpu_manager.go:414] "containerMap: removing container" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.806999 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807006 5109 cpu_manager.go:414] "containerMap: removing container" podUID="658ea0ef-0203-4ad7-982a-274f23cd6d37" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807009 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658ea0ef-0203-4ad7-982a-274f23cd6d37" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807016 5109 cpu_manager.go:414] "containerMap: removing container" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807019 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807027 5109 cpu_manager.go:414] "containerMap: removing container" podUID="5a4b4ea2-9005-4fcf-9de2-29a19050ca73" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807031 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4b4ea2-9005-4fcf-9de2-29a19050ca73" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807039 5109 cpu_manager.go:414] "containerMap: removing container" podUID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807043 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807051 5109 cpu_manager.go:414] "containerMap: removing container" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807055 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807061 5109 cpu_manager.go:414] "containerMap: removing container" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807065 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807072 5109 cpu_manager.go:414] "containerMap: removing container" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807077 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807082 5109 cpu_manager.go:414] "containerMap: removing container" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807086 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807097 5109 cpu_manager.go:414] "containerMap: removing container" podUID="500893a0-a2f3-4840-b9ee-59e5226d9335" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807101 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500893a0-a2f3-4840-b9ee-59e5226d9335" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807155 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb518472-d243-4a01-aead-146570bff41f" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807164 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="658ea0ef-0203-4ad7-982a-274f23cd6d37" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807170 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807176 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4b4ea2-9005-4fcf-9de2-29a19050ca73" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807181 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4ea3c9-b253-455c-b55d-c1f9c2b0b385" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807186 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="50358e22-7a8b-456d-ab7b-41ee21f7c263" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807193 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a45c2e-274a-4f64-acf1-7836436b6c50" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807198 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c12691b-81b3-4967-837c-f32390a02cbb" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807203 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="500893a0-a2f3-4840-b9ee-59e5226d9335" containerName="payload-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807209 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807214 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807220 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ce603b-0b3f-49f8-b801-3ba1d6c952e4" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807226 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe162e44-0b4c-42a3-aacd-2abc635024b8" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807230 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5155c18-83ec-4923-9855-dd0c6f293187" containerName="main" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807236 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8f588d-d144-48c6-8fba-6e6f3a033343" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807241 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807245 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb031617-7de9-4160-ad6b-762744f3c105" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807250 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b9c96c-a9f5-4212-aac8-2379505761d7" containerName="istio-proxy" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807255 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da2ea58-8933-496e-a7d2-7711d131c55d" containerName="payload-pre-processing" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807261 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01b4628-5e7f-486a-9dd2-b5d47cf8df81" containerName="revoke-keys" Nov 02 22:33:51.807236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807266 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="353686ce-7828-478a-b4ad-9f27c3fdc577" containerName="payload-processing" Nov 02 22:33:51.809056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807270 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="205f8467-9b38-454e-b15c-b97e553bee58" containerName="istio-proxy" Nov 02 22:33:51.809056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807275 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="57460eb1-1c58-44dd-b7a8-b2f9ad12e627" containerName="payload-pre-processing" Nov 02 22:33:51.809056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807281 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" containerName="revoke-keys" Nov 02 22:33:51.809056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807286 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="242212e7-a571-4d7d-b880-c6ee0908cdf3" containerName="istio-proxy" Nov 02 22:33:51.809056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807292 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b68edd-abe0-4c44-9f70-c8dfb4dcfcbf" containerName="istio-proxy" Nov 02 22:33:51.809056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807297 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fe2fec-540b-473f-8303-46752a067bdc" containerName="istio-proxy" Nov 02 22:33:51.809056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.807302 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1aa813-c5de-4562-91f7-ea2c64086c30" containerName="istio-proxy" Nov 02 22:33:51.810795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.810778 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.865647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.865614 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv"] Nov 02 22:33:51.874173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.874135 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001dd056-6122-4122-86af-51e86879e299-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.874358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.874261 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qln\" (UniqueName: \"kubernetes.io/projected/001dd056-6122-4122-86af-51e86879e299-kube-api-access-r9qln\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.874437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.874398 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001dd056-6122-4122-86af-51e86879e299-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.912418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.912386 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:33:51.912418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.912403 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:33:51.975153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.975116 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001dd056-6122-4122-86af-51e86879e299-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.975358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.975166 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001dd056-6122-4122-86af-51e86879e299-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.975358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.975223 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qln\" (UniqueName: \"kubernetes.io/projected/001dd056-6122-4122-86af-51e86879e299-kube-api-access-r9qln\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.975977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.975954 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001dd056-6122-4122-86af-51e86879e299-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:51.977642 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:51.977621 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001dd056-6122-4122-86af-51e86879e299-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:52.036200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:52.036164 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qln\" (UniqueName: \"kubernetes.io/projected/001dd056-6122-4122-86af-51e86879e299-kube-api-access-r9qln\") pod \"maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:52.123613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:52.123515 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:52.496777 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:33:52.496741 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001dd056_6122_4122_86af_51e86879e299.slice/crio-254a913e3173c15d9ded3069aa9792666c41cd40aa5c2853aefe719326fa13b2 WatchSource:0}: Error finding container 254a913e3173c15d9ded3069aa9792666c41cd40aa5c2853aefe719326fa13b2: Status 404 returned error can't find the container with id 254a913e3173c15d9ded3069aa9792666c41cd40aa5c2853aefe719326fa13b2 Nov 02 22:33:52.498804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:52.498775 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv"] Nov 02 22:33:53.484161 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:53.484121 5109 generic.go:356] "Generic (PLEG): container finished" podID="001dd056-6122-4122-86af-51e86879e299" containerID="ab50d7ce7729d45c5b2577cb1adfc4d3396b8985957e7a41f8a80862e364824e" exitCode=0 Nov 02 22:33:53.484646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:53.484198 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" event={"ID":"001dd056-6122-4122-86af-51e86879e299","Type":"ContainerDied","Data":"ab50d7ce7729d45c5b2577cb1adfc4d3396b8985957e7a41f8a80862e364824e"} Nov 02 22:33:53.484646 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:53.484254 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" event={"ID":"001dd056-6122-4122-86af-51e86879e299","Type":"ContainerStarted","Data":"254a913e3173c15d9ded3069aa9792666c41cd40aa5c2853aefe719326fa13b2"} Nov 02 22:33:54.628770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.628746 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:54.700712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.700680 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001dd056-6122-4122-86af-51e86879e299-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001dd056-6122-4122-86af-51e86879e299-maas-api-openshift-ca\") pod \"001dd056-6122-4122-86af-51e86879e299\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " Nov 02 22:33:54.700927 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.700727 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001dd056-6122-4122-86af-51e86879e299-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001dd056-6122-4122-86af-51e86879e299-maas-api-xks-ca\") pod \"001dd056-6122-4122-86af-51e86879e299\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " Nov 02 22:33:54.700927 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.700779 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001dd056-6122-4122-86af-51e86879e299-kube-api-access-r9qln\" (UniqueName: \"kubernetes.io/projected/001dd056-6122-4122-86af-51e86879e299-kube-api-access-r9qln\") pod \"001dd056-6122-4122-86af-51e86879e299\" (UID: \"001dd056-6122-4122-86af-51e86879e299\") " Nov 02 22:33:54.701094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.701070 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001dd056-6122-4122-86af-51e86879e299-maas-api-openshift-ca" pod "001dd056-6122-4122-86af-51e86879e299" (UID: "001dd056-6122-4122-86af-51e86879e299"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:54.703083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.703052 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001dd056-6122-4122-86af-51e86879e299-maas-api-xks-ca" pod "001dd056-6122-4122-86af-51e86879e299" (UID: "001dd056-6122-4122-86af-51e86879e299"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:54.703178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.703137 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001dd056-6122-4122-86af-51e86879e299-kube-api-access-r9qln" pod "001dd056-6122-4122-86af-51e86879e299" (UID: "001dd056-6122-4122-86af-51e86879e299"). InnerVolumeSpecName "kube-api-access-r9qln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:54.802397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.802356 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001dd056-6122-4122-86af-51e86879e299-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:54.802397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.802391 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001dd056-6122-4122-86af-51e86879e299-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:54.802397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:54.802402 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9qln\" (UniqueName: \"kubernetes.io/projected/001dd056-6122-4122-86af-51e86879e299-kube-api-access-r9qln\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:55.498698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:55.498668 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" Nov 02 22:33:55.498698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:55.498686 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv" event={"ID":"001dd056-6122-4122-86af-51e86879e299","Type":"ContainerDied","Data":"254a913e3173c15d9ded3069aa9792666c41cd40aa5c2853aefe719326fa13b2"} Nov 02 22:33:55.498896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:33:55.498737 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254a913e3173c15d9ded3069aa9792666c41cd40aa5c2853aefe719326fa13b2" Nov 02 22:33:56.493134 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:33:56.493096 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:34:03.341807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:03.341767 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt"] Nov 02 22:34:03.446820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:03.446786 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn"] Nov 02 22:34:04.051981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:04.051943 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r"] Nov 02 22:34:04.057928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:04.057891 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-445bfe47-mkv5r"] Nov 02 22:34:04.060056 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:04.060033 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59"] Nov 02 22:34:04.061874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:04.061853 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72b6680a-l8n59"] Nov 02 22:34:05.891707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:05.891669 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1632a11f-1152-406e-8201-98615272f92f" path="/var/lib/kubelet/pods/1632a11f-1152-406e-8201-98615272f92f/volumes" Nov 02 22:34:05.892112 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:05.892096 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcd630a-0ac6-4502-9e59-13076d714458" path="/var/lib/kubelet/pods/1bcd630a-0ac6-4502-9e59-13076d714458/volumes" Nov 02 22:34:08.342687 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.342626 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" podUID="89548a9b-fd47-49cd-994b-b45fa6d76fea" containerName="payload-processing" containerID="cri-o://0c148b2a387a53a719f56c23a897ec03277ce89db2c8b6dc1491af138d918a6c" gracePeriod=25 Nov 02 22:34:08.448121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.448066 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" podUID="86194a23-285d-4637-bc7d-9162c7aed560" containerName="payload-pre-processing" containerID="cri-o://6879940abd441664d5380beef99c2120e576453554d5c8617e3588bac03a5afa" gracePeriod=25 Nov 02 22:34:08.584546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.584514 5109 generic.go:356] "Generic (PLEG): container finished" podID="86194a23-285d-4637-bc7d-9162c7aed560" containerID="6879940abd441664d5380beef99c2120e576453554d5c8617e3588bac03a5afa" exitCode=0 Nov 02 22:34:08.584751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.584622 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" event={"ID":"86194a23-285d-4637-bc7d-9162c7aed560","Type":"ContainerDied","Data":"6879940abd441664d5380beef99c2120e576453554d5c8617e3588bac03a5afa"} Nov 02 22:34:08.586291 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.586257 5109 generic.go:356] "Generic (PLEG): container finished" podID="89548a9b-fd47-49cd-994b-b45fa6d76fea" containerID="0c148b2a387a53a719f56c23a897ec03277ce89db2c8b6dc1491af138d918a6c" exitCode=0 Nov 02 22:34:08.586422 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.586304 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" event={"ID":"89548a9b-fd47-49cd-994b-b45fa6d76fea","Type":"ContainerDied","Data":"0c148b2a387a53a719f56c23a897ec03277ce89db2c8b6dc1491af138d918a6c"} Nov 02 22:34:08.611676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.611591 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:34:08.712667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.712641 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:34:08.754972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.754936 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89548a9b-fd47-49cd-994b-b45fa6d76fea-kube-api-access-6dmq4\" (UniqueName: \"kubernetes.io/projected/89548a9b-fd47-49cd-994b-b45fa6d76fea-kube-api-access-6dmq4\") pod \"89548a9b-fd47-49cd-994b-b45fa6d76fea\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " Nov 02 22:34:08.755206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.755122 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89548a9b-fd47-49cd-994b-b45fa6d76fea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89548a9b-fd47-49cd-994b-b45fa6d76fea-plugins-config-volume\") pod \"89548a9b-fd47-49cd-994b-b45fa6d76fea\" (UID: \"89548a9b-fd47-49cd-994b-b45fa6d76fea\") " Nov 02 22:34:08.755506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.755482 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89548a9b-fd47-49cd-994b-b45fa6d76fea-plugins-config-volume" pod "89548a9b-fd47-49cd-994b-b45fa6d76fea" (UID: "89548a9b-fd47-49cd-994b-b45fa6d76fea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:08.757327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.757303 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89548a9b-fd47-49cd-994b-b45fa6d76fea-kube-api-access-6dmq4" pod "89548a9b-fd47-49cd-994b-b45fa6d76fea" (UID: "89548a9b-fd47-49cd-994b-b45fa6d76fea"). InnerVolumeSpecName "kube-api-access-6dmq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:08.856329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.856284 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86194a23-285d-4637-bc7d-9162c7aed560-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86194a23-285d-4637-bc7d-9162c7aed560-plugins-config-volume\") pod \"86194a23-285d-4637-bc7d-9162c7aed560\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " Nov 02 22:34:08.856510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.856418 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86194a23-285d-4637-bc7d-9162c7aed560-kube-api-access-d6ldr\" (UniqueName: \"kubernetes.io/projected/86194a23-285d-4637-bc7d-9162c7aed560-kube-api-access-d6ldr\") pod \"86194a23-285d-4637-bc7d-9162c7aed560\" (UID: \"86194a23-285d-4637-bc7d-9162c7aed560\") " Nov 02 22:34:08.856686 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.856664 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86194a23-285d-4637-bc7d-9162c7aed560-plugins-config-volume" pod "86194a23-285d-4637-bc7d-9162c7aed560" (UID: "86194a23-285d-4637-bc7d-9162c7aed560"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:08.856783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.856770 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmq4\" (UniqueName: \"kubernetes.io/projected/89548a9b-fd47-49cd-994b-b45fa6d76fea-kube-api-access-6dmq4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:08.856826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.856786 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86194a23-285d-4637-bc7d-9162c7aed560-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:08.856826 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.856795 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89548a9b-fd47-49cd-994b-b45fa6d76fea-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:08.858680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.858658 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86194a23-285d-4637-bc7d-9162c7aed560-kube-api-access-d6ldr" pod "86194a23-285d-4637-bc7d-9162c7aed560" (UID: "86194a23-285d-4637-bc7d-9162c7aed560"). InnerVolumeSpecName "kube-api-access-d6ldr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:08.958166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:08.958074 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ldr\" (UniqueName: \"kubernetes.io/projected/86194a23-285d-4637-bc7d-9162c7aed560-kube-api-access-d6ldr\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:09.594460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.594419 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" event={"ID":"86194a23-285d-4637-bc7d-9162c7aed560","Type":"ContainerDied","Data":"36e950e9a15a0a6770d29dafcabccc3f1ecc83f55333a47ebffcff794f7c73b0"} Nov 02 22:34:09.595001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.594511 5109 scope.go:122] "RemoveContainer" containerID="6879940abd441664d5380beef99c2120e576453554d5c8617e3588bac03a5afa" Nov 02 22:34:09.595001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.594432 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn" Nov 02 22:34:09.595979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.595962 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" Nov 02 22:34:09.595979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.595966 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt" event={"ID":"89548a9b-fd47-49cd-994b-b45fa6d76fea","Type":"ContainerDied","Data":"600d07f829650d6002eab7fc45ac3e6c20df3a3b443dfcc7b856355e360f4ebe"} Nov 02 22:34:09.605632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.605609 5109 scope.go:122] "RemoveContainer" containerID="0c148b2a387a53a719f56c23a897ec03277ce89db2c8b6dc1491af138d918a6c" Nov 02 22:34:09.763070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.763040 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt"] Nov 02 22:34:09.769447 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.769417 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb1a7ce7-659fd47955-tdvrt"] Nov 02 22:34:09.880617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.880521 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn"] Nov 02 22:34:09.890429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.890400 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89548a9b-fd47-49cd-994b-b45fa6d76fea" path="/var/lib/kubelet/pods/89548a9b-fd47-49cd-994b-b45fa6d76fea/volumes" Nov 02 22:34:09.890715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:09.890700 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb1a7ce7-7d649959d7-sklvn"] Nov 02 22:34:11.890542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:11.890510 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86194a23-285d-4637-bc7d-9162c7aed560" path="/var/lib/kubelet/pods/86194a23-285d-4637-bc7d-9162c7aed560/volumes" Nov 02 22:34:16.642551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:16.642516 5109 scope.go:122] "RemoveContainer" containerID="43c4115927254ab0cd9b27e4a9d57dbf60eed10a7c0b38bb490d93e1f21c99cc" Nov 02 22:34:16.652557 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:16.652531 5109 scope.go:122] "RemoveContainer" containerID="be2e4d5280433a2047ca5f2c803b1e5f6bd9dc9b2140b535d6148c656a286960" Nov 02 22:34:16.662236 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:16.662215 5109 scope.go:122] "RemoveContainer" containerID="2ec9c1ffc61df737dfda8ed6394e7002cf87afc96a41a4cf60f1409c0603b814" Nov 02 22:34:16.671649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:16.671625 5109 scope.go:122] "RemoveContainer" containerID="cd02433086410a9650da2869f53f9f3d7b83201c9f1acb55ff3d449eb82d93fe" Nov 02 22:34:16.681342 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:16.681323 5109 scope.go:122] "RemoveContainer" containerID="7ac664758e1cdcbbe5d5f6c943b9d508689b15f6663b0245a1dca8c999b46b4c" Nov 02 22:34:16.690171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:16.690147 5109 scope.go:122] "RemoveContainer" containerID="55810f79d9fb9aba2610196bcb5de117478dbda0870cfc43ec509d22d84300aa" Nov 02 22:34:16.699019 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:16.698998 5109 scope.go:122] "RemoveContainer" containerID="78e8de60ecb069a664ba86fa21b8b046315fc1309cad146b56a1ee65e528374e" Nov 02 22:34:17.907342 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:17.907309 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv"] Nov 02 22:34:17.957396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:17.957350 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb1a7ce7-7qtgv"] Nov 02 22:34:19.890998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:19.890967 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001dd056-6122-4122-86af-51e86879e299" path="/var/lib/kubelet/pods/001dd056-6122-4122-86af-51e86879e299/volumes" Nov 02 22:34:25.553202 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553158 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j"] Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553618 5109 cpu_manager.go:414] "containerMap: removing container" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553631 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d2d9c2-5758-4033-a9b8-26e322a0af11" containerName="istio-proxy" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553647 5109 cpu_manager.go:414] "containerMap: removing container" podUID="86194a23-285d-4637-bc7d-9162c7aed560" containerName="payload-pre-processing" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553652 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86194a23-285d-4637-bc7d-9162c7aed560" containerName="payload-pre-processing" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553715 5109 cpu_manager.go:414] "containerMap: removing container" podUID="89548a9b-fd47-49cd-994b-b45fa6d76fea" containerName="payload-processing" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553721 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89548a9b-fd47-49cd-994b-b45fa6d76fea" containerName="payload-processing" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553730 5109 cpu_manager.go:414] "containerMap: removing container" podUID="001dd056-6122-4122-86af-51e86879e299" containerName="revoke-keys" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553734 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001dd056-6122-4122-86af-51e86879e299" containerName="revoke-keys" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553839 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="86194a23-285d-4637-bc7d-9162c7aed560" containerName="payload-pre-processing" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553852 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="89548a9b-fd47-49cd-994b-b45fa6d76fea" containerName="payload-processing" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553858 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="001dd056-6122-4122-86af-51e86879e299" containerName="revoke-keys" Nov 02 22:34:25.553923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.553865 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ec1893-7330-4af0-91d1-448ba3809d8c" containerName="istio-proxy" Nov 02 22:34:25.557166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.557142 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.663247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.663210 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:34:25.663450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.663280 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:34:25.665622 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.665595 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j"] Nov 02 22:34:25.750583 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.750532 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.750780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.750666 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnmdc\" (UniqueName: \"kubernetes.io/projected/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-kube-api-access-mnmdc\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.750780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.750741 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.852198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.852093 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.852198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.852161 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnmdc\" (UniqueName: \"kubernetes.io/projected/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-kube-api-access-mnmdc\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.852412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.852224 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.852950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.852918 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.854677 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.854646 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:25.914214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:25.914175 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnmdc\" (UniqueName: \"kubernetes.io/projected/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-kube-api-access-mnmdc\") pod \"maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:26.173296 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:26.173205 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:26.491069 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:34:26.490978 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:34:26.547738 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:34:26.547699 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fe9d17_3bdb_441c_8bdb_ad38a3fff067.slice/crio-de5e946ad5ab3256ea9a3484f9b0705209b17a424ba1e41641b21cc4486deb26 WatchSource:0}: Error finding container de5e946ad5ab3256ea9a3484f9b0705209b17a424ba1e41641b21cc4486deb26: Status 404 returned error can't find the container with id de5e946ad5ab3256ea9a3484f9b0705209b17a424ba1e41641b21cc4486deb26 Nov 02 22:34:26.549404 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:26.549378 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j"] Nov 02 22:34:26.744247 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:26.744209 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" event={"ID":"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067","Type":"ContainerStarted","Data":"de5e946ad5ab3256ea9a3484f9b0705209b17a424ba1e41641b21cc4486deb26"} Nov 02 22:34:27.751252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:27.751154 5109 generic.go:356] "Generic (PLEG): container finished" podID="d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" containerID="be415ae9db953fdbcb2dac2d80d9cdbb58367d7f007329123efe175d1c2bd8a3" exitCode=0 Nov 02 22:34:27.751676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:27.751242 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" event={"ID":"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067","Type":"ContainerDied","Data":"be415ae9db953fdbcb2dac2d80d9cdbb58367d7f007329123efe175d1c2bd8a3"} Nov 02 22:34:28.901146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:28.901118 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:28.981601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:28.981522 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-xks-ca\") pod \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " Nov 02 22:34:28.981815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:28.981640 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-openshift-ca\") pod \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " Nov 02 22:34:28.981815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:28.981701 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-kube-api-access-mnmdc\" (UniqueName: \"kubernetes.io/projected/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-kube-api-access-mnmdc\") pod \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\" (UID: \"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067\") " Nov 02 22:34:28.982015 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:28.981988 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-openshift-ca" pod "d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" (UID: "d4fe9d17-3bdb-441c-8bdb-ad38a3fff067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:28.983981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:28.983950 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-kube-api-access-mnmdc" pod "d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" (UID: "d4fe9d17-3bdb-441c-8bdb-ad38a3fff067"). InnerVolumeSpecName "kube-api-access-mnmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:28.984090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:28.984008 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-xks-ca" pod "d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" (UID: "d4fe9d17-3bdb-441c-8bdb-ad38a3fff067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:29.082635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:29.082532 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:29.082635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:29.082583 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:29.082635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:29.082597 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnmdc\" (UniqueName: \"kubernetes.io/projected/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067-kube-api-access-mnmdc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:29.766580 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:29.766536 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" Nov 02 22:34:29.766762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:29.766598 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j" event={"ID":"d4fe9d17-3bdb-441c-8bdb-ad38a3fff067","Type":"ContainerDied","Data":"de5e946ad5ab3256ea9a3484f9b0705209b17a424ba1e41641b21cc4486deb26"} Nov 02 22:34:29.766762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:29.766655 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de5e946ad5ab3256ea9a3484f9b0705209b17a424ba1e41641b21cc4486deb26" Nov 02 22:34:36.910536 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:36.910496 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf"] Nov 02 22:34:41.911560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:41.911438 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" podUID="e2cd7261-e112-4be3-936b-7384b02b4889" containerName="payload-processing" containerID="cri-o://c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b" gracePeriod=25 Nov 02 22:34:42.025987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.025947 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg"] Nov 02 22:34:42.026417 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.026405 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" containerName="revoke-keys" Nov 02 22:34:42.026466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.026419 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" containerName="revoke-keys" Nov 02 22:34:42.026506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.026496 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" containerName="revoke-keys" Nov 02 22:34:42.030065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.030048 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.086944 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.086903 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:34:42.099765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.099729 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:34:42.102280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.102256 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj"] Nov 02 22:34:42.106386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.106361 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.113530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.113501 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.113737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.113623 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.113800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.113760 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2dss\" (UniqueName: \"kubernetes.io/projected/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-kube-api-access-k2dss\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.136337 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.136296 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg"] Nov 02 22:34:42.201856 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.201767 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj"] Nov 02 22:34:42.214394 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.214353 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2dss\" (UniqueName: \"kubernetes.io/projected/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-kube-api-access-k2dss\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.214648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.214423 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvrw\" (UniqueName: \"kubernetes.io/projected/ce75bad1-e1c7-4b05-868f-93c0d209b4af-kube-api-access-rnvrw\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.214648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.214452 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.214648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.214545 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.214648 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.214625 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.214865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.214655 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.215516 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.215486 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.217541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.217511 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.281442 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.281400 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2dss\" (UniqueName: \"kubernetes.io/projected/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-kube-api-access-k2dss\") pod \"maas-api-revoke-sub-0f9bcf4d-2j5jg\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.315933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.315896 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvrw\" (UniqueName: \"kubernetes.io/projected/ce75bad1-e1c7-4b05-868f-93c0d209b4af-kube-api-access-rnvrw\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.315933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.315937 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.316195 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.315997 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.316665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.316637 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.318613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.318559 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.343545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.343506 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:42.378911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.378869 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvrw\" (UniqueName: \"kubernetes.io/projected/ce75bad1-e1c7-4b05-868f-93c0d209b4af-kube-api-access-rnvrw\") pod \"maas-api-revoke-sub-fab95197-9cfwj\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.421943 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.421899 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:42.725910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.725882 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg"] Nov 02 22:34:42.727287 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:34:42.727261 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d6f956_f6fc_436c_a143_70cb6ed0e2d8.slice/crio-c738ff714b95ad1f03a0aeabd2e9d650d0ee9b713664d029c4700ad3f8f7cb2f WatchSource:0}: Error finding container c738ff714b95ad1f03a0aeabd2e9d650d0ee9b713664d029c4700ad3f8f7cb2f: Status 404 returned error can't find the container with id c738ff714b95ad1f03a0aeabd2e9d650d0ee9b713664d029c4700ad3f8f7cb2f Nov 02 22:34:42.815292 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:34:42.815227 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce75bad1_e1c7_4b05_868f_93c0d209b4af.slice/crio-2026f89e52649ff41b09fc73356001125c4ffe4527ff4e4db1f9263a5ae3f681 WatchSource:0}: Error finding container 2026f89e52649ff41b09fc73356001125c4ffe4527ff4e4db1f9263a5ae3f681: Status 404 returned error can't find the container with id 2026f89e52649ff41b09fc73356001125c4ffe4527ff4e4db1f9263a5ae3f681 Nov 02 22:34:42.817493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.817466 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj"] Nov 02 22:34:42.843116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.843079 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" event={"ID":"84d6f956-f6fc-436c-a143-70cb6ed0e2d8","Type":"ContainerStarted","Data":"c738ff714b95ad1f03a0aeabd2e9d650d0ee9b713664d029c4700ad3f8f7cb2f"} Nov 02 22:34:42.844399 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:42.844372 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" event={"ID":"ce75bad1-e1c7-4b05-868f-93c0d209b4af","Type":"ContainerStarted","Data":"2026f89e52649ff41b09fc73356001125c4ffe4527ff4e4db1f9263a5ae3f681"} Nov 02 22:34:43.852210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:43.852174 5109 generic.go:356] "Generic (PLEG): container finished" podID="ce75bad1-e1c7-4b05-868f-93c0d209b4af" containerID="767fc7efd35b20285917d199c96a880e47cd01d7077253c4e28792ecf909a3cd" exitCode=0 Nov 02 22:34:43.852749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:43.852268 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" event={"ID":"ce75bad1-e1c7-4b05-868f-93c0d209b4af","Type":"ContainerDied","Data":"767fc7efd35b20285917d199c96a880e47cd01d7077253c4e28792ecf909a3cd"} Nov 02 22:34:43.854000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:43.853979 5109 generic.go:356] "Generic (PLEG): container finished" podID="84d6f956-f6fc-436c-a143-70cb6ed0e2d8" containerID="2a739c320d33e2f3851d5047c86ddc73487e4d8d50011ed04ff6b8f11500fe80" exitCode=0 Nov 02 22:34:43.854099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:43.854024 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" event={"ID":"84d6f956-f6fc-436c-a143-70cb6ed0e2d8","Type":"ContainerDied","Data":"2a739c320d33e2f3851d5047c86ddc73487e4d8d50011ed04ff6b8f11500fe80"} Nov 02 22:34:45.037157 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.037123 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:45.040689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.040665 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:45.148095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.147882 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-openshift-ca\") pod \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " Nov 02 22:34:45.148095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.147976 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-kube-api-access-k2dss\" (UniqueName: \"kubernetes.io/projected/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-kube-api-access-k2dss\") pod \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " Nov 02 22:34:45.148095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.148049 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-xks-ca\") pod \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " Nov 02 22:34:45.148431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.148158 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce75bad1-e1c7-4b05-868f-93c0d209b4af-kube-api-access-rnvrw\" (UniqueName: \"kubernetes.io/projected/ce75bad1-e1c7-4b05-868f-93c0d209b4af-kube-api-access-rnvrw\") pod \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " Nov 02 22:34:45.148431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.148245 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-xks-ca\") pod \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\" (UID: \"84d6f956-f6fc-436c-a143-70cb6ed0e2d8\") " Nov 02 22:34:45.148431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.148314 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-openshift-ca\") pod \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\" (UID: \"ce75bad1-e1c7-4b05-868f-93c0d209b4af\") " Nov 02 22:34:45.148431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.148371 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-openshift-ca" pod "84d6f956-f6fc-436c-a143-70cb6ed0e2d8" (UID: "84d6f956-f6fc-436c-a143-70cb6ed0e2d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:45.148718 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.148702 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.148860 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.148829 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-openshift-ca" pod "ce75bad1-e1c7-4b05-868f-93c0d209b4af" (UID: "ce75bad1-e1c7-4b05-868f-93c0d209b4af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:45.150623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.150589 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce75bad1-e1c7-4b05-868f-93c0d209b4af-kube-api-access-rnvrw" pod "ce75bad1-e1c7-4b05-868f-93c0d209b4af" (UID: "ce75bad1-e1c7-4b05-868f-93c0d209b4af"). InnerVolumeSpecName "kube-api-access-rnvrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:45.150791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.150642 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-kube-api-access-k2dss" pod "84d6f956-f6fc-436c-a143-70cb6ed0e2d8" (UID: "84d6f956-f6fc-436c-a143-70cb6ed0e2d8"). InnerVolumeSpecName "kube-api-access-k2dss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:45.151059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.151035 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-xks-ca" pod "84d6f956-f6fc-436c-a143-70cb6ed0e2d8" (UID: "84d6f956-f6fc-436c-a143-70cb6ed0e2d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:45.151119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.151039 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-xks-ca" pod "ce75bad1-e1c7-4b05-868f-93c0d209b4af" (UID: "ce75bad1-e1c7-4b05-868f-93c0d209b4af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:45.250079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.250040 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.250079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.250075 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2dss\" (UniqueName: \"kubernetes.io/projected/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-kube-api-access-k2dss\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.250079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.250085 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce75bad1-e1c7-4b05-868f-93c0d209b4af-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.250345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.250097 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnvrw\" (UniqueName: \"kubernetes.io/projected/ce75bad1-e1c7-4b05-868f-93c0d209b4af-kube-api-access-rnvrw\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.250345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.250112 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d6f956-f6fc-436c-a143-70cb6ed0e2d8-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.866375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.866339 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" Nov 02 22:34:45.866375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.866365 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg" event={"ID":"84d6f956-f6fc-436c-a143-70cb6ed0e2d8","Type":"ContainerDied","Data":"c738ff714b95ad1f03a0aeabd2e9d650d0ee9b713664d029c4700ad3f8f7cb2f"} Nov 02 22:34:45.866679 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.866446 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c738ff714b95ad1f03a0aeabd2e9d650d0ee9b713664d029c4700ad3f8f7cb2f" Nov 02 22:34:45.867964 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.867940 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" event={"ID":"ce75bad1-e1c7-4b05-868f-93c0d209b4af","Type":"ContainerDied","Data":"2026f89e52649ff41b09fc73356001125c4ffe4527ff4e4db1f9263a5ae3f681"} Nov 02 22:34:45.868103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.867985 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2026f89e52649ff41b09fc73356001125c4ffe4527ff4e4db1f9263a5ae3f681" Nov 02 22:34:45.868103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:45.867950 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj" Nov 02 22:34:51.023946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:51.023902 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" podUID="e2cd7261-e112-4be3-936b-7384b02b4889" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:34:56.490818 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:34:56.490778 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:34:59.554893 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:59.554540 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j"] Nov 02 22:34:59.601678 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:59.601637 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8263a4-z2m6j"] Nov 02 22:34:59.890767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:34:59.890680 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fe9d17-3bdb-441c-8bdb-ad38a3fff067" path="/var/lib/kubelet/pods/d4fe9d17-3bdb-441c-8bdb-ad38a3fff067/volumes" Nov 02 22:35:01.024071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:01.024023 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" podUID="e2cd7261-e112-4be3-936b-7384b02b4889" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:35:02.167129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.167055 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:35:02.332298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.332258 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cd7261-e112-4be3-936b-7384b02b4889-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cd7261-e112-4be3-936b-7384b02b4889-plugins-config-volume\") pod \"e2cd7261-e112-4be3-936b-7384b02b4889\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " Nov 02 22:35:02.332505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.332364 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2cd7261-e112-4be3-936b-7384b02b4889-kube-api-access-jkscq\" (UniqueName: \"kubernetes.io/projected/e2cd7261-e112-4be3-936b-7384b02b4889-kube-api-access-jkscq\") pod \"e2cd7261-e112-4be3-936b-7384b02b4889\" (UID: \"e2cd7261-e112-4be3-936b-7384b02b4889\") " Nov 02 22:35:02.332765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.332737 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cd7261-e112-4be3-936b-7384b02b4889-plugins-config-volume" pod "e2cd7261-e112-4be3-936b-7384b02b4889" (UID: "e2cd7261-e112-4be3-936b-7384b02b4889"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:02.334676 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.334652 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cd7261-e112-4be3-936b-7384b02b4889-kube-api-access-jkscq" pod "e2cd7261-e112-4be3-936b-7384b02b4889" (UID: "e2cd7261-e112-4be3-936b-7384b02b4889"). InnerVolumeSpecName "kube-api-access-jkscq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:02.433914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.433820 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cd7261-e112-4be3-936b-7384b02b4889-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:02.433914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.433856 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkscq\" (UniqueName: \"kubernetes.io/projected/e2cd7261-e112-4be3-936b-7384b02b4889-kube-api-access-jkscq\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:02.970375 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.970339 5109 generic.go:356] "Generic (PLEG): container finished" podID="e2cd7261-e112-4be3-936b-7384b02b4889" containerID="c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b" exitCode=0 Nov 02 22:35:02.970563 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.970387 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" Nov 02 22:35:02.970563 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.970427 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" event={"ID":"e2cd7261-e112-4be3-936b-7384b02b4889","Type":"ContainerDied","Data":"c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b"} Nov 02 22:35:02.970563 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.970486 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf" event={"ID":"e2cd7261-e112-4be3-936b-7384b02b4889","Type":"ContainerDied","Data":"74c5f69137e4788a8b046b976b8d998e68d4b7f432d2fe3e6995b1fe9f384156"} Nov 02 22:35:02.970563 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.970515 5109 scope.go:122] "RemoveContainer" containerID="c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b" Nov 02 22:35:02.981560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.981539 5109 scope.go:122] "RemoveContainer" containerID="c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b" Nov 02 22:35:02.981895 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:35:02.981872 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b\": container with ID starting with c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b not found: ID does not exist" containerID="c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b" Nov 02 22:35:02.981973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:02.981910 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b"} err="failed to get container status \"c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b\": rpc error: code = NotFound desc = could not find container \"c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b\": container with ID starting with c7867c6ce2c83ec452fcf3cc38196fb22d43999b3da4fde56aaf39c35c2d892b not found: ID does not exist" Nov 02 22:35:03.136406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:03.136366 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf"] Nov 02 22:35:03.186245 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:03.186198 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8263a4-6c49cdb675-2jdjf"] Nov 02 22:35:03.891266 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:03.891231 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2cd7261-e112-4be3-936b-7384b02b4889" path="/var/lib/kubelet/pods/e2cd7261-e112-4be3-936b-7384b02b4889/volumes" Nov 02 22:35:04.049284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:04.049250 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l"] Nov 02 22:35:04.099117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:04.099083 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e10957-c4r4l"] Nov 02 22:35:05.890839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:05.890793 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2fce43-c9e9-45dc-9bc5-df008bbb4741" path="/var/lib/kubelet/pods/8b2fce43-c9e9-45dc-9bc5-df008bbb4741/volumes" Nov 02 22:35:07.540192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:07.540149 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66"] Nov 02 22:35:07.540727 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:07.540384 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" containerID="cri-o://065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a" gracePeriod=30 Nov 02 22:35:12.795229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.795201 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:35:12.839823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839737 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-data\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.839823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839778 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9cf88a-65ba-4646-bd38-179e399737db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9cf88a-65ba-4646-bd38-179e399737db-istiod-ca-cert\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839835 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-socket\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839858 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-envoy\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839888 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-certs\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839913 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-istio-token\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-istio-token\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839950 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-credential-socket\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.839988 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-kube-api-access-ldbkn\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-kube-api-access-ldbkn\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840030 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb9cf88a-65ba-4646-bd38-179e399737db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9cf88a-65ba-4646-bd38-179e399737db-istio-podinfo\") pod \"bb9cf88a-65ba-4646-bd38-179e399737db\" (UID: \"bb9cf88a-65ba-4646-bd38-179e399737db\") " Nov 02 22:35:12.840106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840070 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-data" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.840560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840140 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-socket" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.840560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840223 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-certs" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.840560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840312 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9cf88a-65ba-4646-bd38-179e399737db-istiod-ca-cert" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:12.840560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840402 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.840560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840425 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9cf88a-65ba-4646-bd38-179e399737db-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.840560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840440 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.840560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840452 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.840910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.840698 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-credential-socket" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.842427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.842398 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-envoy" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.842427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.842401 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-istio-token" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.842664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.842502 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb9cf88a-65ba-4646-bd38-179e399737db-istio-podinfo" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:35:12.842839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.842820 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-kube-api-access-ldbkn" pod "bb9cf88a-65ba-4646-bd38-179e399737db" (UID: "bb9cf88a-65ba-4646-bd38-179e399737db"). InnerVolumeSpecName "kube-api-access-ldbkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.941928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.941881 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.941928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.941911 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.941928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.941921 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9cf88a-65ba-4646-bd38-179e399737db-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.941928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.941931 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldbkn\" (UniqueName: \"kubernetes.io/projected/bb9cf88a-65ba-4646-bd38-179e399737db-kube-api-access-ldbkn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.941928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:12.941940 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9cf88a-65ba-4646-bd38-179e399737db-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.042153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.042120 5109 generic.go:356] "Generic (PLEG): container finished" podID="bb9cf88a-65ba-4646-bd38-179e399737db" containerID="065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a" exitCode=0 Nov 02 22:35:13.042343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.042189 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" event={"ID":"bb9cf88a-65ba-4646-bd38-179e399737db","Type":"ContainerDied","Data":"065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a"} Nov 02 22:35:13.042343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.042195 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" Nov 02 22:35:13.042343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.042232 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66" event={"ID":"bb9cf88a-65ba-4646-bd38-179e399737db","Type":"ContainerDied","Data":"77f39e8afa4c4da89f2ca610102c08c8809069a4198cb5b6e80b8fd49625e13a"} Nov 02 22:35:13.042343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.042253 5109 scope.go:122] "RemoveContainer" containerID="065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a" Nov 02 22:35:13.052716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.052546 5109 scope.go:122] "RemoveContainer" containerID="065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a" Nov 02 22:35:13.052959 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:35:13.052938 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a\": container with ID starting with 065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a not found: ID does not exist" containerID="065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a" Nov 02 22:35:13.053028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.052967 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a"} err="failed to get container status \"065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a\": rpc error: code = NotFound desc = could not find container \"065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a\": container with ID starting with 065314e121123b0afc57f519172dd160bec6004103a9ec029bf39448a084de4a not found: ID does not exist" Nov 02 22:35:13.211819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.211733 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66"] Nov 02 22:35:13.215877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.215848 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-zfx66"] Nov 02 22:35:13.897385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:13.897349 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" path="/var/lib/kubelet/pods/bb9cf88a-65ba-4646-bd38-179e399737db/volumes" Nov 02 22:35:16.805970 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:16.805930 5109 scope.go:122] "RemoveContainer" containerID="d75f81e94848f00f604f944392dfc5a14c76b9e9bbbbb163020c01b5bfb6fdb3" Nov 02 22:35:16.815419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:16.815399 5109 scope.go:122] "RemoveContainer" containerID="0b474e647ebe44c0e3bbeb0905d7931ee4c96be1040c9c8ba3b5c73b4ec15b69" Nov 02 22:35:16.824223 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:16.824199 5109 scope.go:122] "RemoveContainer" containerID="09708276ee637b40bb9fec61f475d3a6e214038b7114bbf148ef5641b35f84ce" Nov 02 22:35:16.833956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:16.833935 5109 scope.go:122] "RemoveContainer" containerID="3522ad772dcc8a1d0ac2472ee880ffaa2091e908095e660ef82086fc29514fa9" Nov 02 22:35:22.765439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:22.765398 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-nd9wf"] Nov 02 22:35:22.765881 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:22.765656 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" podUID="805d005f-e957-4d8b-a6d9-6978b429c1d6" containerName="manager" containerID="cri-o://547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd" gracePeriod=30 Nov 02 22:35:23.010982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.010949 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:35:23.043509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.043429 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805d005f-e957-4d8b-a6d9-6978b429c1d6-kube-api-access-gbmjp\" (UniqueName: \"kubernetes.io/projected/805d005f-e957-4d8b-a6d9-6978b429c1d6-kube-api-access-gbmjp\") pod \"805d005f-e957-4d8b-a6d9-6978b429c1d6\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " Nov 02 22:35:23.043697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.043513 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-metrics-certs\") pod \"805d005f-e957-4d8b-a6d9-6978b429c1d6\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " Nov 02 22:35:23.043697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.043656 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-webhook-cert\") pod \"805d005f-e957-4d8b-a6d9-6978b429c1d6\" (UID: \"805d005f-e957-4d8b-a6d9-6978b429c1d6\") " Nov 02 22:35:23.045818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.045786 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-metrics-certs" pod "805d005f-e957-4d8b-a6d9-6978b429c1d6" (UID: "805d005f-e957-4d8b-a6d9-6978b429c1d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:23.045939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.045848 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-webhook-cert" pod "805d005f-e957-4d8b-a6d9-6978b429c1d6" (UID: "805d005f-e957-4d8b-a6d9-6978b429c1d6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:23.045939 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.045903 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805d005f-e957-4d8b-a6d9-6978b429c1d6-kube-api-access-gbmjp" pod "805d005f-e957-4d8b-a6d9-6978b429c1d6" (UID: "805d005f-e957-4d8b-a6d9-6978b429c1d6"). InnerVolumeSpecName "kube-api-access-gbmjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:23.102072 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.102030 5109 generic.go:356] "Generic (PLEG): container finished" podID="805d005f-e957-4d8b-a6d9-6978b429c1d6" containerID="547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd" exitCode=0 Nov 02 22:35:23.102264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.102081 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" Nov 02 22:35:23.102264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.102122 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" event={"ID":"805d005f-e957-4d8b-a6d9-6978b429c1d6","Type":"ContainerDied","Data":"547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd"} Nov 02 22:35:23.102264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.102186 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-nd9wf" event={"ID":"805d005f-e957-4d8b-a6d9-6978b429c1d6","Type":"ContainerDied","Data":"e5c1c0227cc074d821b55365b75baa8dade1b8905b499274b331ecfad646e65a"} Nov 02 22:35:23.102264 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.102211 5109 scope.go:122] "RemoveContainer" containerID="547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd" Nov 02 22:35:23.116496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.116474 5109 scope.go:122] "RemoveContainer" containerID="547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd" Nov 02 22:35:23.116895 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:35:23.116871 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd\": container with ID starting with 547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd not found: ID does not exist" containerID="547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd" Nov 02 22:35:23.117011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.116903 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd"} err="failed to get container status \"547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd\": rpc error: code = NotFound desc = could not find container \"547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd\": container with ID starting with 547da7f7d6998cd00d3aa96d74659c21cd1686081e527ef12131af38abf07ddd not found: ID does not exist" Nov 02 22:35:23.145096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.145060 5109 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-metrics-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:23.145096 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.145093 5109 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/805d005f-e957-4d8b-a6d9-6978b429c1d6-webhook-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:23.145371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.145106 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmjp\" (UniqueName: \"kubernetes.io/projected/805d005f-e957-4d8b-a6d9-6978b429c1d6-kube-api-access-gbmjp\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:23.245020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.244987 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-nd9wf"] Nov 02 22:35:23.247278 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.247246 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-nd9wf"] Nov 02 22:35:23.891760 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:23.891726 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805d005f-e957-4d8b-a6d9-6978b429c1d6" path="/var/lib/kubelet/pods/805d005f-e957-4d8b-a6d9-6978b429c1d6/volumes" Nov 02 22:35:24.076870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.076828 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-546bfbd88f-2r6j4"] Nov 02 22:35:24.077244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077232 5109 cpu_manager.go:414] "containerMap: removing container" podUID="805d005f-e957-4d8b-a6d9-6978b429c1d6" containerName="manager" Nov 02 22:35:24.077293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077246 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805d005f-e957-4d8b-a6d9-6978b429c1d6" containerName="manager" Nov 02 22:35:24.077293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077256 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e2cd7261-e112-4be3-936b-7384b02b4889" containerName="payload-processing" Nov 02 22:35:24.077293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077262 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cd7261-e112-4be3-936b-7384b02b4889" containerName="payload-processing" Nov 02 22:35:24.077293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077283 5109 cpu_manager.go:414] "containerMap: removing container" podUID="84d6f956-f6fc-436c-a143-70cb6ed0e2d8" containerName="revoke-keys" Nov 02 22:35:24.077293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077288 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d6f956-f6fc-436c-a143-70cb6ed0e2d8" containerName="revoke-keys" Nov 02 22:35:24.077293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077293 5109 cpu_manager.go:414] "containerMap: removing container" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077298 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077312 5109 cpu_manager.go:414] "containerMap: removing container" podUID="ce75bad1-e1c7-4b05-868f-93c0d209b4af" containerName="revoke-keys" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077317 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce75bad1-e1c7-4b05-868f-93c0d209b4af" containerName="revoke-keys" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077383 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce75bad1-e1c7-4b05-868f-93c0d209b4af" containerName="revoke-keys" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077395 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="805d005f-e957-4d8b-a6d9-6978b429c1d6" containerName="manager" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077401 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cd7261-e112-4be3-936b-7384b02b4889" containerName="payload-processing" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077409 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d6f956-f6fc-436c-a143-70cb6ed0e2d8" containerName="revoke-keys" Nov 02 22:35:24.077479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.077415 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9cf88a-65ba-4646-bd38-179e399737db" containerName="istio-proxy" Nov 02 22:35:24.081981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.081961 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.134921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.134892 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:35:24.155819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.155728 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-metrics-certs\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.155819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.155801 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-webhook-cert\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.156011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.155903 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/11f6a33b-21c1-466a-9c1f-0089faa62f5a-kube-api-access-hlhbm\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.184775 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.184740 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:35:24.184973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.184819 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:35:24.184973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.184842 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jsb8q\"" Nov 02 22:35:24.188758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.188731 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-2r6j4"] Nov 02 22:35:24.257101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.257065 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-webhook-cert\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.257295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.257121 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/11f6a33b-21c1-466a-9c1f-0089faa62f5a-kube-api-access-hlhbm\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.257295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.257167 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-metrics-certs\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.259851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.259820 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-webhook-cert\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.259978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.259939 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-metrics-certs\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.318980 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.318946 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/11f6a33b-21c1-466a-9c1f-0089faa62f5a-kube-api-access-hlhbm\") pod \"maas-controller-546bfbd88f-2r6j4\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.395460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.395412 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:24.774312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.774273 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-2r6j4"] Nov 02 22:35:24.802494 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:35:24.802460 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f6a33b_21c1_466a_9c1f_0089faa62f5a.slice/crio-7c37894c3efd8461d70101e197913ac6277e788328f7d782e07f846afa8fa468 WatchSource:0}: Error finding container 7c37894c3efd8461d70101e197913ac6277e788328f7d782e07f846afa8fa468: Status 404 returned error can't find the container with id 7c37894c3efd8461d70101e197913ac6277e788328f7d782e07f846afa8fa468 Nov 02 22:35:24.804008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:24.803986 5109 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:35:25.117861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:25.117748 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" event={"ID":"11f6a33b-21c1-466a-9c1f-0089faa62f5a","Type":"ContainerStarted","Data":"7c37894c3efd8461d70101e197913ac6277e788328f7d782e07f846afa8fa468"} Nov 02 22:35:26.140637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:26.131389 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" event={"ID":"11f6a33b-21c1-466a-9c1f-0089faa62f5a","Type":"ContainerStarted","Data":"08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad"} Nov 02 22:35:26.140637 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:26.132457 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:26.519941 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:35:26.519906 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:35:27.465728 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:27.465682 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:35:27.467055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:27.466875 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:35:38.162888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:38.162857 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:38.281556 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:38.281494 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" podStartSLOduration=13.315982491 podStartE2EDuration="14.281480694s" totalImagesPullingTime="965.498203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:26.262302396 +0000 UTC m=+1762.527295615" watchObservedRunningTime="2026-11-02 22:35:38.281480694 +0000 UTC m=+1774.546473913" Nov 02 22:35:43.940465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:43.940425 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2"] Nov 02 22:35:43.944284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:43.944260 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:43.999665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:43.999630 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2"] Nov 02 22:35:44.046956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.046918 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:35:44.047148 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.046977 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:35:44.069126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.069084 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.069368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.069268 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88j76\" (UniqueName: \"kubernetes.io/projected/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-kube-api-access-88j76\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.069368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.069304 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.170765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.170722 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88j76\" (UniqueName: \"kubernetes.io/projected/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-kube-api-access-88j76\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.170765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.170768 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.170994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.170810 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.171508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.171474 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.173479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.173460 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.232174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.232088 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88j76\" (UniqueName: \"kubernetes.io/projected/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-kube-api-access-88j76\") pod \"maas-api-revoke-sub-85237e45-9jpm2\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.258005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.257969 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:44.647331 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:35:44.647288 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a65cc06_3b3a_40a3_b8c2_0fdd056cd592.slice/crio-ff0c1b05bd2daba8a708fb80d180de2bce88ea37d74282cc24e7fc12a60dc216 WatchSource:0}: Error finding container ff0c1b05bd2daba8a708fb80d180de2bce88ea37d74282cc24e7fc12a60dc216: Status 404 returned error can't find the container with id ff0c1b05bd2daba8a708fb80d180de2bce88ea37d74282cc24e7fc12a60dc216 Nov 02 22:35:44.650018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:44.649992 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2"] Nov 02 22:35:45.263907 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:45.263872 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" event={"ID":"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592","Type":"ContainerStarted","Data":"727a221e5cbf20f1be4b671d197409d6d660a7fa2e7800892b9b1922b878bfd0"} Nov 02 22:35:45.264345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:45.263927 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" event={"ID":"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592","Type":"ContainerStarted","Data":"ff0c1b05bd2daba8a708fb80d180de2bce88ea37d74282cc24e7fc12a60dc216"} Nov 02 22:35:45.381436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:45.381374 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" podStartSLOduration=2.381357802 podStartE2EDuration="2.381357802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:45.379635324 +0000 UTC m=+1781.644628544" watchObservedRunningTime="2026-11-02 22:35:45.381357802 +0000 UTC m=+1781.646351021" Nov 02 22:35:46.270762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:46.270728 5109 generic.go:356] "Generic (PLEG): container finished" podID="4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" containerID="727a221e5cbf20f1be4b671d197409d6d660a7fa2e7800892b9b1922b878bfd0" exitCode=0 Nov 02 22:35:46.271190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:46.270813 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" event={"ID":"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592","Type":"ContainerDied","Data":"727a221e5cbf20f1be4b671d197409d6d660a7fa2e7800892b9b1922b878bfd0"} Nov 02 22:35:47.418499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.418469 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:47.503137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.503095 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-xks-ca\") pod \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " Nov 02 22:35:47.503137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.503142 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-openshift-ca\") pod \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " Nov 02 22:35:47.503422 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.503206 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-kube-api-access-88j76\" (UniqueName: \"kubernetes.io/projected/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-kube-api-access-88j76\") pod \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\" (UID: \"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592\") " Nov 02 22:35:47.503539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.503514 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-openshift-ca" pod "4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" (UID: "4a65cc06-3b3a-40a3-b8c2-0fdd056cd592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:47.505520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.505494 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-xks-ca" pod "4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" (UID: "4a65cc06-3b3a-40a3-b8c2-0fdd056cd592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:47.505629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.505553 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-kube-api-access-88j76" pod "4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" (UID: "4a65cc06-3b3a-40a3-b8c2-0fdd056cd592"). InnerVolumeSpecName "kube-api-access-88j76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:47.604428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.604334 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88j76\" (UniqueName: \"kubernetes.io/projected/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-kube-api-access-88j76\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:47.604428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.604371 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:47.604428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:47.604384 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:48.290967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:48.290929 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" event={"ID":"4a65cc06-3b3a-40a3-b8c2-0fdd056cd592","Type":"ContainerDied","Data":"ff0c1b05bd2daba8a708fb80d180de2bce88ea37d74282cc24e7fc12a60dc216"} Nov 02 22:35:48.290967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:48.290964 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2" Nov 02 22:35:48.291201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:48.291006 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0c1b05bd2daba8a708fb80d180de2bce88ea37d74282cc24e7fc12a60dc216" Nov 02 22:35:56.306664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.306625 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-2r6j4"] Nov 02 22:35:56.307123 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.306844 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" podUID="11f6a33b-21c1-466a-9c1f-0089faa62f5a" containerName="manager" containerID="cri-o://08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad" gracePeriod=30 Nov 02 22:35:56.498402 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:35:56.498359 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:35:56.560010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.559941 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:56.694259 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.694227 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-webhook-cert\") pod \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " Nov 02 22:35:56.694456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.694285 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-metrics-certs\") pod \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " Nov 02 22:35:56.694456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.694366 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f6a33b-21c1-466a-9c1f-0089faa62f5a-kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/11f6a33b-21c1-466a-9c1f-0089faa62f5a-kube-api-access-hlhbm\") pod \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\" (UID: \"11f6a33b-21c1-466a-9c1f-0089faa62f5a\") " Nov 02 22:35:56.696973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.696938 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-metrics-certs" pod "11f6a33b-21c1-466a-9c1f-0089faa62f5a" (UID: "11f6a33b-21c1-466a-9c1f-0089faa62f5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:56.696973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.696941 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f6a33b-21c1-466a-9c1f-0089faa62f5a-kube-api-access-hlhbm" pod "11f6a33b-21c1-466a-9c1f-0089faa62f5a" (UID: "11f6a33b-21c1-466a-9c1f-0089faa62f5a"). InnerVolumeSpecName "kube-api-access-hlhbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:56.697250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.697228 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-webhook-cert" pod "11f6a33b-21c1-466a-9c1f-0089faa62f5a" (UID: "11f6a33b-21c1-466a-9c1f-0089faa62f5a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:56.796135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.796092 5109 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-metrics-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:56.796135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.796132 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/11f6a33b-21c1-466a-9c1f-0089faa62f5a-kube-api-access-hlhbm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:56.796358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:56.796147 5109 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11f6a33b-21c1-466a-9c1f-0089faa62f5a-webhook-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:57.345011 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.344976 5109 generic.go:356] "Generic (PLEG): container finished" podID="11f6a33b-21c1-466a-9c1f-0089faa62f5a" containerID="08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad" exitCode=0 Nov 02 22:35:57.345466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.345025 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" Nov 02 22:35:57.345466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.345068 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" event={"ID":"11f6a33b-21c1-466a-9c1f-0089faa62f5a","Type":"ContainerDied","Data":"08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad"} Nov 02 22:35:57.345466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.345129 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-2r6j4" event={"ID":"11f6a33b-21c1-466a-9c1f-0089faa62f5a","Type":"ContainerDied","Data":"7c37894c3efd8461d70101e197913ac6277e788328f7d782e07f846afa8fa468"} Nov 02 22:35:57.345466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.345152 5109 scope.go:122] "RemoveContainer" containerID="08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad" Nov 02 22:35:57.355845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.355823 5109 scope.go:122] "RemoveContainer" containerID="08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad" Nov 02 22:35:57.356149 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:35:57.356118 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad\": container with ID starting with 08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad not found: ID does not exist" containerID="08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad" Nov 02 22:35:57.356200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.356157 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad"} err="failed to get container status \"08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad\": rpc error: code = NotFound desc = could not find container \"08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad\": container with ID starting with 08b7def100e4a5056202e2b4a7261c894516833e7cbd292354cdf38cd5ead2ad not found: ID does not exist" Nov 02 22:35:57.509814 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.509774 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-2r6j4"] Nov 02 22:35:57.559792 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.559751 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-2r6j4"] Nov 02 22:35:57.890636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:57.890599 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f6a33b-21c1-466a-9c1f-0089faa62f5a" path="/var/lib/kubelet/pods/11f6a33b-21c1-466a-9c1f-0089faa62f5a/volumes" Nov 02 22:35:58.961125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.961084 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-546bfbd88f-9cdcm"] Nov 02 22:35:58.961580 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.961547 5109 cpu_manager.go:414] "containerMap: removing container" podUID="4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" containerName="revoke-keys" Nov 02 22:35:58.961580 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.961577 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" containerName="revoke-keys" Nov 02 22:35:58.961712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.961599 5109 cpu_manager.go:414] "containerMap: removing container" podUID="11f6a33b-21c1-466a-9c1f-0089faa62f5a" containerName="manager" Nov 02 22:35:58.961712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.961604 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f6a33b-21c1-466a-9c1f-0089faa62f5a" containerName="manager" Nov 02 22:35:58.961712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.961666 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" containerName="revoke-keys" Nov 02 22:35:58.961712 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.961676 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f6a33b-21c1-466a-9c1f-0089faa62f5a" containerName="manager" Nov 02 22:35:58.966331 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:58.966307 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.069079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.069049 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:35:59.069600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.069547 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jsb8q\"" Nov 02 22:35:59.069600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.069555 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:35:59.069801 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.069606 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:35:59.072111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.072088 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv"] Nov 02 22:35:59.074206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.074182 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-229108f6-nmwtv"] Nov 02 22:35:59.076052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.076031 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-9cdcm"] Nov 02 22:35:59.120016 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.119981 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8783ce4a-aa7d-4a16-aa4f-7cd954884912-webhook-cert\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.120205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.120114 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59k8w\" (UniqueName: \"kubernetes.io/projected/8783ce4a-aa7d-4a16-aa4f-7cd954884912-kube-api-access-59k8w\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.120205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.120163 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8783ce4a-aa7d-4a16-aa4f-7cd954884912-metrics-certs\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.221681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.221597 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8783ce4a-aa7d-4a16-aa4f-7cd954884912-webhook-cert\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.221681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.221652 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59k8w\" (UniqueName: \"kubernetes.io/projected/8783ce4a-aa7d-4a16-aa4f-7cd954884912-kube-api-access-59k8w\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.221681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.221681 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8783ce4a-aa7d-4a16-aa4f-7cd954884912-metrics-certs\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.224395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.224366 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8783ce4a-aa7d-4a16-aa4f-7cd954884912-metrics-certs\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.224527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.224366 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8783ce4a-aa7d-4a16-aa4f-7cd954884912-webhook-cert\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.283859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.283818 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59k8w\" (UniqueName: \"kubernetes.io/projected/8783ce4a-aa7d-4a16-aa4f-7cd954884912-kube-api-access-59k8w\") pod \"maas-controller-546bfbd88f-9cdcm\" (UID: \"8783ce4a-aa7d-4a16-aa4f-7cd954884912\") " pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.580000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.579942 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:35:59.891091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.890991 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0751d6d8-d378-4ad7-b34f-3452368f57ee" path="/var/lib/kubelet/pods/0751d6d8-d378-4ad7-b34f-3452368f57ee/volumes" Nov 02 22:35:59.953632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:35:59.953592 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-546bfbd88f-9cdcm"] Nov 02 22:35:59.980635 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:35:59.980593 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8783ce4a_aa7d_4a16_aa4f_7cd954884912.slice/crio-3d9ad9d6aac434bab1303b36fca33669a38b5aac71292785c6bb9911c67d8b87 WatchSource:0}: Error finding container 3d9ad9d6aac434bab1303b36fca33669a38b5aac71292785c6bb9911c67d8b87: Status 404 returned error can't find the container with id 3d9ad9d6aac434bab1303b36fca33669a38b5aac71292785c6bb9911c67d8b87 Nov 02 22:36:00.364937 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:00.364889 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-9cdcm" event={"ID":"8783ce4a-aa7d-4a16-aa4f-7cd954884912","Type":"ContainerStarted","Data":"3d9ad9d6aac434bab1303b36fca33669a38b5aac71292785c6bb9911c67d8b87"} Nov 02 22:36:01.373412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:01.373375 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-546bfbd88f-9cdcm" event={"ID":"8783ce4a-aa7d-4a16-aa4f-7cd954884912","Type":"ContainerStarted","Data":"bbcd3c2f36fdaffcbe35af7f6839a7e353eefbdf0075031d5ab1a6bbd83c67a3"} Nov 02 22:36:01.373989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:01.373578 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:36:01.492423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:01.492369 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-546bfbd88f-9cdcm" podStartSLOduration=2.611046391 podStartE2EDuration="3.492353047s" totalImagesPullingTime="881.306656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:01.492108468 +0000 UTC m=+1797.757101686" watchObservedRunningTime="2026-11-02 22:36:01.492353047 +0000 UTC m=+1797.757346265" Nov 02 22:36:12.382070 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:12.381982 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-546bfbd88f-9cdcm" Nov 02 22:36:16.877046 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:16.877013 5109 scope.go:122] "RemoveContainer" containerID="811ca4c78285f2407f7cc7592137b03c1bc8cadb03b986ff39efb1fb5ab9f2b6" Nov 02 22:36:16.886495 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:16.886469 5109 scope.go:122] "RemoveContainer" containerID="e6859f3b1c2a8bd406678118c60e8ec82fb5d41f4d1c8bb2de25a70ca6bc1241" Nov 02 22:36:16.895491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:16.895463 5109 scope.go:122] "RemoveContainer" containerID="b46fdf031a70f4ffcd5a320f68cdb85686e6865687df54e468893880f882d839" Nov 02 22:36:16.904457 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:16.904435 5109 scope.go:122] "RemoveContainer" containerID="4822713d4b3d32c1874e328d8f7843cc1a9c7ba61b6ed808c379f38a9311bde8" Nov 02 22:36:26.490802 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:36:26.490750 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:36:54.293993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.293950 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb"] Nov 02 22:36:54.297462 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.297443 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.358546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.358510 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb"] Nov 02 22:36:54.381031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.380989 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88f8ead-e8d2-4191-82db-355962649063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.381225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.381102 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/d88f8ead-e8d2-4191-82db-355962649063-kube-api-access-lhztf\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.381225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.381170 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88f8ead-e8d2-4191-82db-355962649063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.405981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.405943 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:36:54.405981 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.405973 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:36:54.481941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.481900 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/d88f8ead-e8d2-4191-82db-355962649063-kube-api-access-lhztf\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.481941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.481947 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88f8ead-e8d2-4191-82db-355962649063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.482190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.482089 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88f8ead-e8d2-4191-82db-355962649063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.482818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.482794 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88f8ead-e8d2-4191-82db-355962649063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.484605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.484583 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88f8ead-e8d2-4191-82db-355962649063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.543282 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.543249 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/d88f8ead-e8d2-4191-82db-355962649063-kube-api-access-lhztf\") pod \"maas-api-revoke-sub-c1e271a9-kzjqb\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.610976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.610872 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:54.979811 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:36:54.979777 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88f8ead_e8d2_4191_82db_355962649063.slice/crio-94ca3ea9f88adec9183450a421436ebc7d876c85f9922a4e16d9e2842fd3e74e WatchSource:0}: Error finding container 94ca3ea9f88adec9183450a421436ebc7d876c85f9922a4e16d9e2842fd3e74e: Status 404 returned error can't find the container with id 94ca3ea9f88adec9183450a421436ebc7d876c85f9922a4e16d9e2842fd3e74e Nov 02 22:36:54.981240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:54.981211 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb"] Nov 02 22:36:55.705414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:55.705379 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" event={"ID":"d88f8ead-e8d2-4191-82db-355962649063","Type":"ContainerStarted","Data":"f9e713b8b090ddf51f1b501628f57937081bea8a721f4d862d413001afba0048"} Nov 02 22:36:55.705878 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:55.705441 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" event={"ID":"d88f8ead-e8d2-4191-82db-355962649063","Type":"ContainerStarted","Data":"94ca3ea9f88adec9183450a421436ebc7d876c85f9922a4e16d9e2842fd3e74e"} Nov 02 22:36:55.823828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:55.823768 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" podStartSLOduration=1.8237511400000002 podStartE2EDuration="1.82375114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:55.821163306 +0000 UTC m=+1852.086156525" watchObservedRunningTime="2026-11-02 22:36:55.82375114 +0000 UTC m=+1852.088744358" Nov 02 22:36:56.490761 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:36:56.490722 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:36:56.713240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:56.713200 5109 generic.go:356] "Generic (PLEG): container finished" podID="d88f8ead-e8d2-4191-82db-355962649063" containerID="f9e713b8b090ddf51f1b501628f57937081bea8a721f4d862d413001afba0048" exitCode=0 Nov 02 22:36:56.713658 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:56.713253 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" event={"ID":"d88f8ead-e8d2-4191-82db-355962649063","Type":"ContainerDied","Data":"f9e713b8b090ddf51f1b501628f57937081bea8a721f4d862d413001afba0048"} Nov 02 22:36:57.860423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:57.860389 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:57.909262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:57.909223 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d88f8ead-e8d2-4191-82db-355962649063-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88f8ead-e8d2-4191-82db-355962649063-maas-api-xks-ca\") pod \"d88f8ead-e8d2-4191-82db-355962649063\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " Nov 02 22:36:57.909262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:57.909271 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88f8ead-e8d2-4191-82db-355962649063-kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/d88f8ead-e8d2-4191-82db-355962649063-kube-api-access-lhztf\") pod \"d88f8ead-e8d2-4191-82db-355962649063\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " Nov 02 22:36:57.909541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:57.909317 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88f8ead-e8d2-4191-82db-355962649063-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88f8ead-e8d2-4191-82db-355962649063-maas-api-openshift-ca\") pod \"d88f8ead-e8d2-4191-82db-355962649063\" (UID: \"d88f8ead-e8d2-4191-82db-355962649063\") " Nov 02 22:36:57.909754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:57.909723 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88f8ead-e8d2-4191-82db-355962649063-maas-api-openshift-ca" pod "d88f8ead-e8d2-4191-82db-355962649063" (UID: "d88f8ead-e8d2-4191-82db-355962649063"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:36:57.911613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:57.911586 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88f8ead-e8d2-4191-82db-355962649063-kube-api-access-lhztf" pod "d88f8ead-e8d2-4191-82db-355962649063" (UID: "d88f8ead-e8d2-4191-82db-355962649063"). InnerVolumeSpecName "kube-api-access-lhztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:57.911724 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:57.911686 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88f8ead-e8d2-4191-82db-355962649063-maas-api-xks-ca" pod "d88f8ead-e8d2-4191-82db-355962649063" (UID: "d88f8ead-e8d2-4191-82db-355962649063"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:58.010059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:58.010026 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88f8ead-e8d2-4191-82db-355962649063-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:58.010059 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:58.010061 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhztf\" (UniqueName: \"kubernetes.io/projected/d88f8ead-e8d2-4191-82db-355962649063-kube-api-access-lhztf\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:58.010283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:58.010073 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88f8ead-e8d2-4191-82db-355962649063-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:58.726805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:58.726774 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" Nov 02 22:36:58.726996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:58.726776 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb" event={"ID":"d88f8ead-e8d2-4191-82db-355962649063","Type":"ContainerDied","Data":"94ca3ea9f88adec9183450a421436ebc7d876c85f9922a4e16d9e2842fd3e74e"} Nov 02 22:36:58.726996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:36:58.726900 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94ca3ea9f88adec9183450a421436ebc7d876c85f9922a4e16d9e2842fd3e74e" Nov 02 22:37:06.050668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:06.050633 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw"] Nov 02 22:37:06.099489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:06.099446 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx"] Nov 02 22:37:06.101482 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:06.101459 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f74292b7-ddtqw"] Nov 02 22:37:06.103191 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:06.103167 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bedd8c4-498jx"] Nov 02 22:37:07.891335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:07.891299 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54750126-b5a6-4de7-82a3-b32c71bc6f45" path="/var/lib/kubelet/pods/54750126-b5a6-4de7-82a3-b32c71bc6f45/volumes" Nov 02 22:37:07.891771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:07.891741 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f3878b-4a2a-4689-9a7b-0455af40d585" path="/var/lib/kubelet/pods/92f3878b-4a2a-4689-9a7b-0455af40d585/volumes" Nov 02 22:37:16.949787 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:16.949751 5109 scope.go:122] "RemoveContainer" containerID="4aac5f378868c76703b3f5735e34488f9af8d8aeced5c7e5f8e2199d1903bfc0" Nov 02 22:37:25.750736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.750699 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp"] Nov 02 22:37:25.751243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.751134 5109 cpu_manager.go:414] "containerMap: removing container" podUID="d88f8ead-e8d2-4191-82db-355962649063" containerName="revoke-keys" Nov 02 22:37:25.751243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.751146 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88f8ead-e8d2-4191-82db-355962649063" containerName="revoke-keys" Nov 02 22:37:25.751243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.751219 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88f8ead-e8d2-4191-82db-355962649063" containerName="revoke-keys" Nov 02 22:37:25.754493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.754476 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.774946 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.774906 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spj9k\" (UniqueName: \"kubernetes.io/projected/3352cfbd-c5c7-4686-83c9-eb9905b51d38-kube-api-access-spj9k\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.775130 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.775101 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.775184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.775147 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.810823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.810783 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp"] Nov 02 22:37:25.857590 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.857525 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:25.857789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.857656 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:37:25.875705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.875672 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.875905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.875721 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.875905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.875753 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spj9k\" (UniqueName: \"kubernetes.io/projected/3352cfbd-c5c7-4686-83c9-eb9905b51d38-kube-api-access-spj9k\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.876437 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.876410 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.878397 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.878369 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:25.938087 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:25.938055 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spj9k\" (UniqueName: \"kubernetes.io/projected/3352cfbd-c5c7-4686-83c9-eb9905b51d38-kube-api-access-spj9k\") pod \"maas-api-revoke-sub-2e1f5b5f-mn4kp\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:26.067730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:26.067680 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:26.443918 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:37:26.443883 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3352cfbd_c5c7_4686_83c9_eb9905b51d38.slice/crio-3375839abf3d90cc92958c00651776112ecb97f55422e126421f4f7676433dab WatchSource:0}: Error finding container 3375839abf3d90cc92958c00651776112ecb97f55422e126421f4f7676433dab: Status 404 returned error can't find the container with id 3375839abf3d90cc92958c00651776112ecb97f55422e126421f4f7676433dab Nov 02 22:37:26.445178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:26.445154 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp"] Nov 02 22:37:26.496291 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:37:26.496254 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:37:26.888908 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:26.888864 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" event={"ID":"3352cfbd-c5c7-4686-83c9-eb9905b51d38","Type":"ContainerStarted","Data":"5e1c9e482f7f60b4ba4abaf467686fd277b1be8eb900b70408d273f725104177"} Nov 02 22:37:26.889405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:26.888920 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" event={"ID":"3352cfbd-c5c7-4686-83c9-eb9905b51d38","Type":"ContainerStarted","Data":"3375839abf3d90cc92958c00651776112ecb97f55422e126421f4f7676433dab"} Nov 02 22:37:27.005978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:27.005924 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" podStartSLOduration=2.005907855 podStartE2EDuration="2.005907855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:27.004668095 +0000 UTC m=+1883.269661316" watchObservedRunningTime="2026-11-02 22:37:27.005907855 +0000 UTC m=+1883.270901075" Nov 02 22:37:27.895448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:27.895402 5109 generic.go:356] "Generic (PLEG): container finished" podID="3352cfbd-c5c7-4686-83c9-eb9905b51d38" containerID="5e1c9e482f7f60b4ba4abaf467686fd277b1be8eb900b70408d273f725104177" exitCode=0 Nov 02 22:37:27.895913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:27.895485 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" event={"ID":"3352cfbd-c5c7-4686-83c9-eb9905b51d38","Type":"ContainerDied","Data":"5e1c9e482f7f60b4ba4abaf467686fd277b1be8eb900b70408d273f725104177"} Nov 02 22:37:29.041891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.041867 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:29.106713 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.106676 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-xks-ca\") pod \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " Nov 02 22:37:29.106928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.106726 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3352cfbd-c5c7-4686-83c9-eb9905b51d38-kube-api-access-spj9k\" (UniqueName: \"kubernetes.io/projected/3352cfbd-c5c7-4686-83c9-eb9905b51d38-kube-api-access-spj9k\") pod \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " Nov 02 22:37:29.106928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.106804 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-openshift-ca\") pod \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\" (UID: \"3352cfbd-c5c7-4686-83c9-eb9905b51d38\") " Nov 02 22:37:29.107252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.107226 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-openshift-ca" pod "3352cfbd-c5c7-4686-83c9-eb9905b51d38" (UID: "3352cfbd-c5c7-4686-83c9-eb9905b51d38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:37:29.109500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.109460 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-xks-ca" pod "3352cfbd-c5c7-4686-83c9-eb9905b51d38" (UID: "3352cfbd-c5c7-4686-83c9-eb9905b51d38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:37:29.117507 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.117474 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3352cfbd-c5c7-4686-83c9-eb9905b51d38-kube-api-access-spj9k" pod "3352cfbd-c5c7-4686-83c9-eb9905b51d38" (UID: "3352cfbd-c5c7-4686-83c9-eb9905b51d38"). InnerVolumeSpecName "kube-api-access-spj9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:37:29.208305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.208206 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:29.208305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.208258 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spj9k\" (UniqueName: \"kubernetes.io/projected/3352cfbd-c5c7-4686-83c9-eb9905b51d38-kube-api-access-spj9k\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:29.208305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.208277 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3352cfbd-c5c7-4686-83c9-eb9905b51d38-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:29.907949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.907920 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" Nov 02 22:37:29.907949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.907930 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp" event={"ID":"3352cfbd-c5c7-4686-83c9-eb9905b51d38","Type":"ContainerDied","Data":"3375839abf3d90cc92958c00651776112ecb97f55422e126421f4f7676433dab"} Nov 02 22:37:29.908159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:29.907985 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3375839abf3d90cc92958c00651776112ecb97f55422e126421f4f7676433dab" Nov 02 22:37:45.051245 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:45.051164 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg"] Nov 02 22:37:45.102314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:45.102282 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286"] Nov 02 22:37:45.105628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:45.105610 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fb8ae3-fcxfg"] Nov 02 22:37:45.108889 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:45.108855 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e240e2-49286"] Nov 02 22:37:45.890960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:45.890924 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee07c24-deb2-4296-a3ad-c7bede1c7d92" path="/var/lib/kubelet/pods/7ee07c24-deb2-4296-a3ad-c7bede1c7d92/volumes" Nov 02 22:37:45.891334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:45.891321 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe" path="/var/lib/kubelet/pods/8bd3c167-4d7c-42c6-bf4b-9ce510fcbafe/volumes" Nov 02 22:37:48.190125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.190085 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx"] Nov 02 22:37:48.190545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.190509 5109 cpu_manager.go:414] "containerMap: removing container" podUID="3352cfbd-c5c7-4686-83c9-eb9905b51d38" containerName="revoke-keys" Nov 02 22:37:48.190545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.190520 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3352cfbd-c5c7-4686-83c9-eb9905b51d38" containerName="revoke-keys" Nov 02 22:37:48.190807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.190625 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="3352cfbd-c5c7-4686-83c9-eb9905b51d38" containerName="revoke-keys" Nov 02 22:37:48.194936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.194916 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.248733 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.248701 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx"] Nov 02 22:37:48.296475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.296433 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:48.296699 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.296495 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:37:48.309172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.309129 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzwj\" (UniqueName: \"kubernetes.io/projected/13ae9319-0ed0-44b3-b252-7cdce332b086-kube-api-access-kfzwj\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.309359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.309300 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.309414 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.309382 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.410832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.410794 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzwj\" (UniqueName: \"kubernetes.io/projected/13ae9319-0ed0-44b3-b252-7cdce332b086-kube-api-access-kfzwj\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.411053 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.410856 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.411053 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.410892 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.411668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.411646 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.413538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.413513 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.472062 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.471974 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzwj\" (UniqueName: \"kubernetes.io/projected/13ae9319-0ed0-44b3-b252-7cdce332b086-kube-api-access-kfzwj\") pod \"maas-api-revoke-sub-ec843018-xzqjx\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.508845 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.508806 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:48.882610 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:37:48.882545 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ae9319_0ed0_44b3_b252_7cdce332b086.slice/crio-9c48c73ca3442f4b1be965e2e917f4068df9b95bdd9ae6eba5eb1bba05f3b4b9 WatchSource:0}: Error finding container 9c48c73ca3442f4b1be965e2e917f4068df9b95bdd9ae6eba5eb1bba05f3b4b9: Status 404 returned error can't find the container with id 9c48c73ca3442f4b1be965e2e917f4068df9b95bdd9ae6eba5eb1bba05f3b4b9 Nov 02 22:37:48.884252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:48.884204 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx"] Nov 02 22:37:49.025089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:49.025043 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" event={"ID":"13ae9319-0ed0-44b3-b252-7cdce332b086","Type":"ContainerStarted","Data":"9c48c73ca3442f4b1be965e2e917f4068df9b95bdd9ae6eba5eb1bba05f3b4b9"} Nov 02 22:37:50.033692 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:50.033645 5109 generic.go:356] "Generic (PLEG): container finished" podID="13ae9319-0ed0-44b3-b252-7cdce332b086" containerID="efcefd6d9143732ebb405665c8ed3c9bc7bb371dcdaa71c080fd321be3b96c58" exitCode=0 Nov 02 22:37:50.034117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:50.033692 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" event={"ID":"13ae9319-0ed0-44b3-b252-7cdce332b086","Type":"ContainerDied","Data":"efcefd6d9143732ebb405665c8ed3c9bc7bb371dcdaa71c080fd321be3b96c58"} Nov 02 22:37:51.187926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.187894 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:51.341029 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.340946 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ae9319-0ed0-44b3-b252-7cdce332b086-kube-api-access-kfzwj\" (UniqueName: \"kubernetes.io/projected/13ae9319-0ed0-44b3-b252-7cdce332b086-kube-api-access-kfzwj\") pod \"13ae9319-0ed0-44b3-b252-7cdce332b086\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " Nov 02 22:37:51.341220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.341123 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-openshift-ca\") pod \"13ae9319-0ed0-44b3-b252-7cdce332b086\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " Nov 02 22:37:51.341220 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.341162 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-xks-ca\") pod \"13ae9319-0ed0-44b3-b252-7cdce332b086\" (UID: \"13ae9319-0ed0-44b3-b252-7cdce332b086\") " Nov 02 22:37:51.341467 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.341444 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-openshift-ca" pod "13ae9319-0ed0-44b3-b252-7cdce332b086" (UID: "13ae9319-0ed0-44b3-b252-7cdce332b086"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:37:51.343184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.343153 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ae9319-0ed0-44b3-b252-7cdce332b086-kube-api-access-kfzwj" pod "13ae9319-0ed0-44b3-b252-7cdce332b086" (UID: "13ae9319-0ed0-44b3-b252-7cdce332b086"). InnerVolumeSpecName "kube-api-access-kfzwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:37:51.343400 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.343378 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-xks-ca" pod "13ae9319-0ed0-44b3-b252-7cdce332b086" (UID: "13ae9319-0ed0-44b3-b252-7cdce332b086"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:37:51.442325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.442288 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:51.442325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.442324 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfzwj\" (UniqueName: \"kubernetes.io/projected/13ae9319-0ed0-44b3-b252-7cdce332b086-kube-api-access-kfzwj\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:51.442549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:51.442337 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13ae9319-0ed0-44b3-b252-7cdce332b086-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:52.047555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:52.047521 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" Nov 02 22:37:52.047555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:52.047541 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx" event={"ID":"13ae9319-0ed0-44b3-b252-7cdce332b086","Type":"ContainerDied","Data":"9c48c73ca3442f4b1be965e2e917f4068df9b95bdd9ae6eba5eb1bba05f3b4b9"} Nov 02 22:37:52.047803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:37:52.047609 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c48c73ca3442f4b1be965e2e917f4068df9b95bdd9ae6eba5eb1bba05f3b4b9" Nov 02 22:37:56.490760 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:37:56.490714 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:38:04.057362 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:04.057324 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8"] Nov 02 22:38:04.107647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:04.107612 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51342356-s9kc8"] Nov 02 22:38:05.054895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.054856 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc"] Nov 02 22:38:05.104890 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.104851 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-074e9677-87msc"] Nov 02 22:38:05.113843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.113791 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9"] Nov 02 22:38:05.114119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.113920 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4"] Nov 02 22:38:05.114949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.114927 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cb776d-xrgr9"] Nov 02 22:38:05.154593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.154548 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d300a2-vn2w4"] Nov 02 22:38:05.894430 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.894386 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2cb4bc-34ca-4fbb-918d-003b7b7d8690" path="/var/lib/kubelet/pods/1d2cb4bc-34ca-4fbb-918d-003b7b7d8690/volumes" Nov 02 22:38:05.895061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.895038 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658ea0ef-0203-4ad7-982a-274f23cd6d37" path="/var/lib/kubelet/pods/658ea0ef-0203-4ad7-982a-274f23cd6d37/volumes" Nov 02 22:38:05.895633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.895611 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01b4628-5e7f-486a-9dd2-b5d47cf8df81" path="/var/lib/kubelet/pods/c01b4628-5e7f-486a-9dd2-b5d47cf8df81/volumes" Nov 02 22:38:05.896700 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:05.896672 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59aff8e-0f7e-4217-921a-ccb7a5e24ac3" path="/var/lib/kubelet/pods/f59aff8e-0f7e-4217-921a-ccb7a5e24ac3/volumes" Nov 02 22:38:16.973200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:16.973169 5109 scope.go:122] "RemoveContainer" containerID="9857895186e8c40345dbc0780a16b5b7074fcec96643fc852e843640685307d5" Nov 02 22:38:16.982650 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:16.982620 5109 scope.go:122] "RemoveContainer" containerID="7a6d50c21bda1289f322d7b2e290e459cce1349d18248cb8522861690dbde457" Nov 02 22:38:16.991459 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:16.991439 5109 scope.go:122] "RemoveContainer" containerID="2ebd1a27a3ea037960ff5352ebbaa139dc83aa0da521af3a6d24949b8b4f07bd" Nov 02 22:38:16.999626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:16.999597 5109 scope.go:122] "RemoveContainer" containerID="a090c6c0b26843b3f459765af8a75818610dd370cbb6895faec5f82398e75a2f" Nov 02 22:38:17.007768 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:17.007747 5109 scope.go:122] "RemoveContainer" containerID="d3bd07e868420ab2a99ff2bc1dc3afcc8c9918e1c8c1a687f7cbf14ad5eb1c97" Nov 02 22:38:17.015840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:17.015820 5109 scope.go:122] "RemoveContainer" containerID="145ec5e76c99cd5164b92740bf56753822976048597784a088cdbce710e4e88f" Nov 02 22:38:23.185859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.185807 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks"] Nov 02 22:38:23.186873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.186845 5109 cpu_manager.go:414] "containerMap: removing container" podUID="13ae9319-0ed0-44b3-b252-7cdce332b086" containerName="revoke-keys" Nov 02 22:38:23.186873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.186873 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ae9319-0ed0-44b3-b252-7cdce332b086" containerName="revoke-keys" Nov 02 22:38:23.187129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.187115 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ae9319-0ed0-44b3-b252-7cdce332b086" containerName="revoke-keys" Nov 02 22:38:23.191288 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.191215 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.250129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.250094 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks"] Nov 02 22:38:23.273101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.273056 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.273292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.273110 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72xbf\" (UniqueName: \"kubernetes.io/projected/727ee4e1-f320-4ae3-b8eb-d085936af589-kube-api-access-72xbf\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.273292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.273253 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.296938 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.296900 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:38:23.297117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.296939 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:38:23.374833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.374790 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.374833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.374837 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72xbf\" (UniqueName: \"kubernetes.io/projected/727ee4e1-f320-4ae3-b8eb-d085936af589-kube-api-access-72xbf\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.375119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.374893 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.375701 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.375676 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.377532 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.377508 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.436924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.436847 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72xbf\" (UniqueName: \"kubernetes.io/projected/727ee4e1-f320-4ae3-b8eb-d085936af589-kube-api-access-72xbf\") pod \"maas-api-revoke-sub-046e1343-c5vks\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.512796 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.512734 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:23.885924 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:38:23.885874 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727ee4e1_f320_4ae3_b8eb_d085936af589.slice/crio-15f3d8ac7ac4471a6c877b766af97ae885fc87b27ac9c1dce96ce2a4b8f9b01c WatchSource:0}: Error finding container 15f3d8ac7ac4471a6c877b766af97ae885fc87b27ac9c1dce96ce2a4b8f9b01c: Status 404 returned error can't find the container with id 15f3d8ac7ac4471a6c877b766af97ae885fc87b27ac9c1dce96ce2a4b8f9b01c Nov 02 22:38:23.891465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:23.891441 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks"] Nov 02 22:38:24.259639 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:24.259595 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" event={"ID":"727ee4e1-f320-4ae3-b8eb-d085936af589","Type":"ContainerStarted","Data":"74fa02739f3de024acabbb958509edba7a3ec2b36c5c6cd221b505f105b68df7"} Nov 02 22:38:24.260058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:24.259663 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" event={"ID":"727ee4e1-f320-4ae3-b8eb-d085936af589","Type":"ContainerStarted","Data":"15f3d8ac7ac4471a6c877b766af97ae885fc87b27ac9c1dce96ce2a4b8f9b01c"} Nov 02 22:38:24.377758 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:24.377694 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" podStartSLOduration=1.377676546 podStartE2EDuration="1.377676546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:24.375187748 +0000 UTC m=+1940.640180972" watchObservedRunningTime="2026-11-02 22:38:24.377676546 +0000 UTC m=+1940.642669764" Nov 02 22:38:25.267001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:25.266964 5109 generic.go:356] "Generic (PLEG): container finished" podID="727ee4e1-f320-4ae3-b8eb-d085936af589" containerID="74fa02739f3de024acabbb958509edba7a3ec2b36c5c6cd221b505f105b68df7" exitCode=0 Nov 02 22:38:25.267492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:25.267048 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" event={"ID":"727ee4e1-f320-4ae3-b8eb-d085936af589","Type":"ContainerDied","Data":"74fa02739f3de024acabbb958509edba7a3ec2b36c5c6cd221b505f105b68df7"} Nov 02 22:38:26.408422 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.408395 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:26.490552 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:38:26.490514 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:38:26.505555 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.505526 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727ee4e1-f320-4ae3-b8eb-d085936af589-kube-api-access-72xbf\" (UniqueName: \"kubernetes.io/projected/727ee4e1-f320-4ae3-b8eb-d085936af589-kube-api-access-72xbf\") pod \"727ee4e1-f320-4ae3-b8eb-d085936af589\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " Nov 02 22:38:26.505746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.505585 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-xks-ca\") pod \"727ee4e1-f320-4ae3-b8eb-d085936af589\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " Nov 02 22:38:26.505746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.505709 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-openshift-ca\") pod \"727ee4e1-f320-4ae3-b8eb-d085936af589\" (UID: \"727ee4e1-f320-4ae3-b8eb-d085936af589\") " Nov 02 22:38:26.506017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.505991 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-openshift-ca" pod "727ee4e1-f320-4ae3-b8eb-d085936af589" (UID: "727ee4e1-f320-4ae3-b8eb-d085936af589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:26.506138 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.506123 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:26.507841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.507812 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-xks-ca" pod "727ee4e1-f320-4ae3-b8eb-d085936af589" (UID: "727ee4e1-f320-4ae3-b8eb-d085936af589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:26.507955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.507935 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727ee4e1-f320-4ae3-b8eb-d085936af589-kube-api-access-72xbf" pod "727ee4e1-f320-4ae3-b8eb-d085936af589" (UID: "727ee4e1-f320-4ae3-b8eb-d085936af589"). InnerVolumeSpecName "kube-api-access-72xbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:26.607355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.607256 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72xbf\" (UniqueName: \"kubernetes.io/projected/727ee4e1-f320-4ae3-b8eb-d085936af589-kube-api-access-72xbf\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:26.607355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:26.607295 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/727ee4e1-f320-4ae3-b8eb-d085936af589-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:27.279530 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:27.279501 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" Nov 02 22:38:27.279737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:27.279546 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks" event={"ID":"727ee4e1-f320-4ae3-b8eb-d085936af589","Type":"ContainerDied","Data":"15f3d8ac7ac4471a6c877b766af97ae885fc87b27ac9c1dce96ce2a4b8f9b01c"} Nov 02 22:38:27.279737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:27.279608 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f3d8ac7ac4471a6c877b766af97ae885fc87b27ac9c1dce96ce2a4b8f9b01c" Nov 02 22:38:33.968221 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:33.968184 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz"] Nov 02 22:38:33.968683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:33.968649 5109 cpu_manager.go:414] "containerMap: removing container" podUID="727ee4e1-f320-4ae3-b8eb-d085936af589" containerName="revoke-keys" Nov 02 22:38:33.968683 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:33.968662 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727ee4e1-f320-4ae3-b8eb-d085936af589" containerName="revoke-keys" Nov 02 22:38:33.968771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:33.968728 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="727ee4e1-f320-4ae3-b8eb-d085936af589" containerName="revoke-keys" Nov 02 22:38:33.971926 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:33.971908 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.025694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.025662 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:38:34.028102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.028044 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz"] Nov 02 22:38:34.074911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.074876 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:38:34.084494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.084463 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.084688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.084528 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.084766 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.084744 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nn9g\" (UniqueName: \"kubernetes.io/projected/c6d1ec81-1928-4772-9204-79e9cd939434-kube-api-access-8nn9g\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.185511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.185474 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn9g\" (UniqueName: \"kubernetes.io/projected/c6d1ec81-1928-4772-9204-79e9cd939434-kube-api-access-8nn9g\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.185511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.185524 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.185834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.185600 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.186307 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.186285 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.188163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.188141 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.247358 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.247271 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nn9g\" (UniqueName: \"kubernetes.io/projected/c6d1ec81-1928-4772-9204-79e9cd939434-kube-api-access-8nn9g\") pod \"maas-api-revoke-sub-d3fba056-k4mtz\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.284594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.284520 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:34.658483 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:34.658448 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz"] Nov 02 22:38:34.661583 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:38:34.661540 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d1ec81_1928_4772_9204_79e9cd939434.slice/crio-1185b8cbca67bab24941a06204e80fbfc5e38ab691a7e37a30d7ffc029a21c84 WatchSource:0}: Error finding container 1185b8cbca67bab24941a06204e80fbfc5e38ab691a7e37a30d7ffc029a21c84: Status 404 returned error can't find the container with id 1185b8cbca67bab24941a06204e80fbfc5e38ab691a7e37a30d7ffc029a21c84 Nov 02 22:38:35.325349 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:35.325300 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" event={"ID":"c6d1ec81-1928-4772-9204-79e9cd939434","Type":"ContainerStarted","Data":"342056b255bf67dc25c851e5016a4406ef8367e8d4737c9893b6bfafd7a16658"} Nov 02 22:38:35.325882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:35.325363 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" event={"ID":"c6d1ec81-1928-4772-9204-79e9cd939434","Type":"ContainerStarted","Data":"1185b8cbca67bab24941a06204e80fbfc5e38ab691a7e37a30d7ffc029a21c84"} Nov 02 22:38:35.444903 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:35.444844 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" podStartSLOduration=2.444828771 podStartE2EDuration="2.444828771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:35.443814894 +0000 UTC m=+1951.708808113" watchObservedRunningTime="2026-11-02 22:38:35.444828771 +0000 UTC m=+1951.709822049" Nov 02 22:38:36.332526 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:36.332484 5109 generic.go:356] "Generic (PLEG): container finished" podID="c6d1ec81-1928-4772-9204-79e9cd939434" containerID="342056b255bf67dc25c851e5016a4406ef8367e8d4737c9893b6bfafd7a16658" exitCode=0 Nov 02 22:38:36.332993 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:36.332664 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" event={"ID":"c6d1ec81-1928-4772-9204-79e9cd939434","Type":"ContainerDied","Data":"342056b255bf67dc25c851e5016a4406ef8367e8d4737c9893b6bfafd7a16658"} Nov 02 22:38:37.789149 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.788802 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:37.821675 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.821625 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-xks-ca\") pod \"c6d1ec81-1928-4772-9204-79e9cd939434\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " Nov 02 22:38:37.821675 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.821685 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d1ec81-1928-4772-9204-79e9cd939434-kube-api-access-8nn9g\" (UniqueName: \"kubernetes.io/projected/c6d1ec81-1928-4772-9204-79e9cd939434-kube-api-access-8nn9g\") pod \"c6d1ec81-1928-4772-9204-79e9cd939434\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " Nov 02 22:38:37.821989 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.821830 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-openshift-ca\") pod \"c6d1ec81-1928-4772-9204-79e9cd939434\" (UID: \"c6d1ec81-1928-4772-9204-79e9cd939434\") " Nov 02 22:38:37.822180 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.822160 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-openshift-ca" pod "c6d1ec81-1928-4772-9204-79e9cd939434" (UID: "c6d1ec81-1928-4772-9204-79e9cd939434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:37.836040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.824040 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d1ec81-1928-4772-9204-79e9cd939434-kube-api-access-8nn9g" pod "c6d1ec81-1928-4772-9204-79e9cd939434" (UID: "c6d1ec81-1928-4772-9204-79e9cd939434"). InnerVolumeSpecName "kube-api-access-8nn9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:37.859613 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.854838 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-xks-ca" pod "c6d1ec81-1928-4772-9204-79e9cd939434" (UID: "c6d1ec81-1928-4772-9204-79e9cd939434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:37.922823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.922791 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nn9g\" (UniqueName: \"kubernetes.io/projected/c6d1ec81-1928-4772-9204-79e9cd939434-kube-api-access-8nn9g\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:37.922823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.922824 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:37.923058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:37.922841 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d1ec81-1928-4772-9204-79e9cd939434-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:38.395299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:38.393145 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" event={"ID":"c6d1ec81-1928-4772-9204-79e9cd939434","Type":"ContainerDied","Data":"1185b8cbca67bab24941a06204e80fbfc5e38ab691a7e37a30d7ffc029a21c84"} Nov 02 22:38:38.395299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:38.393256 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1185b8cbca67bab24941a06204e80fbfc5e38ab691a7e37a30d7ffc029a21c84" Nov 02 22:38:38.395299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:38:38.393358 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz" Nov 02 22:38:56.489771 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:38:56.489723 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:39:12.402682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.402585 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5"] Nov 02 22:39:12.403237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.403041 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c6d1ec81-1928-4772-9204-79e9cd939434" containerName="revoke-keys" Nov 02 22:39:12.403237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.403056 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d1ec81-1928-4772-9204-79e9cd939434" containerName="revoke-keys" Nov 02 22:39:12.403237 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.403149 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d1ec81-1928-4772-9204-79e9cd939434" containerName="revoke-keys" Nov 02 22:39:12.406352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.406332 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.454743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.454700 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.454743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.454744 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4k2x\" (UniqueName: \"kubernetes.io/projected/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-kube-api-access-x4k2x\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.455017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.454848 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.462729 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.462639 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:39:12.464542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.464510 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5"] Nov 02 22:39:12.511836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.511801 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:39:12.556362 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.556323 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.556561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.556367 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4k2x\" (UniqueName: \"kubernetes.io/projected/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-kube-api-access-x4k2x\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.556561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.556412 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.557162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.557142 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.558924 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.558900 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.617846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.617814 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4k2x\" (UniqueName: \"kubernetes.io/projected/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-kube-api-access-x4k2x\") pod \"maas-api-revoke-sub-ae594979-pl6h5\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:12.719730 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:12.719641 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:13.089354 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:39:13.089314 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba6c883_4208_4ec7_abc0_9f8ce0df0b39.slice/crio-57e36d73cade7907315ea79bb71080a2278bb5d1b79483b6ab0fd440bd9c6cb7 WatchSource:0}: Error finding container 57e36d73cade7907315ea79bb71080a2278bb5d1b79483b6ab0fd440bd9c6cb7: Status 404 returned error can't find the container with id 57e36d73cade7907315ea79bb71080a2278bb5d1b79483b6ab0fd440bd9c6cb7 Nov 02 22:39:13.091682 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:13.091654 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5"] Nov 02 22:39:13.607309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:13.607205 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" event={"ID":"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39","Type":"ContainerStarted","Data":"bedef08c3c346824457a45477ef1cf7a0be596803a49f39ba81fb942ee4d5b47"} Nov 02 22:39:13.607309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:13.607264 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" event={"ID":"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39","Type":"ContainerStarted","Data":"57e36d73cade7907315ea79bb71080a2278bb5d1b79483b6ab0fd440bd9c6cb7"} Nov 02 22:39:13.724008 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:13.723953 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" podStartSLOduration=1.7239356190000001 podStartE2EDuration="1.723935619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:13.722497575 +0000 UTC m=+1989.987490793" watchObservedRunningTime="2026-11-02 22:39:13.723935619 +0000 UTC m=+1989.988928837" Nov 02 22:39:14.613995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:14.613956 5109 generic.go:356] "Generic (PLEG): container finished" podID="1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" containerID="bedef08c3c346824457a45477ef1cf7a0be596803a49f39ba81fb942ee4d5b47" exitCode=0 Nov 02 22:39:14.614407 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:14.614054 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" event={"ID":"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39","Type":"ContainerDied","Data":"bedef08c3c346824457a45477ef1cf7a0be596803a49f39ba81fb942ee4d5b47"} Nov 02 22:39:15.761923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.761896 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:15.890498 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.890413 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-openshift-ca\") pod \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " Nov 02 22:39:15.890654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.890545 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-xks-ca\") pod \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " Nov 02 22:39:15.890654 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.890644 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-kube-api-access-x4k2x\" (UniqueName: \"kubernetes.io/projected/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-kube-api-access-x4k2x\") pod \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\" (UID: \"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39\") " Nov 02 22:39:15.890851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.890822 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-openshift-ca" pod "1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" (UID: "1ba6c883-4208-4ec7-abc0-9f8ce0df0b39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:39:15.892923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.892895 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-xks-ca" pod "1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" (UID: "1ba6c883-4208-4ec7-abc0-9f8ce0df0b39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:39:15.893017 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.892987 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-kube-api-access-x4k2x" pod "1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" (UID: "1ba6c883-4208-4ec7-abc0-9f8ce0df0b39"). InnerVolumeSpecName "kube-api-access-x4k2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:39:15.991448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.991414 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:15.991448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.991443 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4k2x\" (UniqueName: \"kubernetes.io/projected/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-kube-api-access-x4k2x\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:15.991448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:15.991455 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:16.626274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:16.626243 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" Nov 02 22:39:16.626500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:16.626283 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5" event={"ID":"1ba6c883-4208-4ec7-abc0-9f8ce0df0b39","Type":"ContainerDied","Data":"57e36d73cade7907315ea79bb71080a2278bb5d1b79483b6ab0fd440bd9c6cb7"} Nov 02 22:39:16.626500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:16.626335 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57e36d73cade7907315ea79bb71080a2278bb5d1b79483b6ab0fd440bd9c6cb7" Nov 02 22:39:17.065139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.065103 5109 scope.go:122] "RemoveContainer" containerID="21a828df79592611cd1bac0abd12bfda69935d43dfea84cd1bfd610034d3373b" Nov 02 22:39:17.074972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.074951 5109 scope.go:122] "RemoveContainer" containerID="26bee02fababa08ab45613007818fa70afe58b449a91a59816a91e25520d4250" Nov 02 22:39:17.083953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.083932 5109 scope.go:122] "RemoveContainer" containerID="6289035c83b4e99421c64bd78e6605141206a519d2d51854fb4948d04e868311" Nov 02 22:39:17.092754 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.092732 5109 scope.go:122] "RemoveContainer" containerID="813c26b2d8099449bef43e1bfddfab24f3692641c96a89185827500111380f6a" Nov 02 22:39:17.101084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.101065 5109 scope.go:122] "RemoveContainer" containerID="a8fb90947179ae3c93c3b958df050178775f0777db2d416bb2272867617c5e43" Nov 02 22:39:17.109833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.109807 5109 scope.go:122] "RemoveContainer" containerID="af44dd9261c0996827b5e7d4a668ef071d3e8ef2994920fbe36b5f6b575e40b1" Nov 02 22:39:17.118521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.118500 5109 scope.go:122] "RemoveContainer" containerID="f2482fcc22ac91fd48556f964a8c10c1830284cca2da4e6aad42f9e6927d4baf" Nov 02 22:39:17.127205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.127184 5109 scope.go:122] "RemoveContainer" containerID="79b8a04a5ec1054a6620842f4a21380c21b413fb86ab1295c2e55368bcaec43f" Nov 02 22:39:17.135915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:17.135895 5109 scope.go:122] "RemoveContainer" containerID="7cefc66e78c2e5109f470c20209e5730fd4cf9b6597999be0b401dc2ec1324b4" Nov 02 22:39:26.490740 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:39:26.490699 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:39:40.584677 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.584644 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x"] Nov 02 22:39:40.585131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.585059 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" containerName="revoke-keys" Nov 02 22:39:40.585131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.585071 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" containerName="revoke-keys" Nov 02 22:39:40.585206 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.585158 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" containerName="revoke-keys" Nov 02 22:39:40.588380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.588363 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.683911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.683878 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.683911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.683922 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.684153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.684045 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5g4\" (UniqueName: \"kubernetes.io/projected/f63c140b-171d-4193-b2ec-50ca0e30005d-kube-api-access-tm5g4\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.691286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.691256 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:39:40.691286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.691278 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:39:40.693346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.693324 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x"] Nov 02 22:39:40.784608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.784545 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.784794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.784661 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5g4\" (UniqueName: \"kubernetes.io/projected/f63c140b-171d-4193-b2ec-50ca0e30005d-kube-api-access-tm5g4\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.784794 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.784761 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.785182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.785162 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.787297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.787279 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.846415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.846336 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5g4\" (UniqueName: \"kubernetes.io/projected/f63c140b-171d-4193-b2ec-50ca0e30005d-kube-api-access-tm5g4\") pod \"maas-api-revoke-sub-a4aef7d7-2rv2x\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:40.901653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:40.901598 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:41.275185 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:39:41.275149 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63c140b_171d_4193_b2ec_50ca0e30005d.slice/crio-c26696d8b416e02bf37169f835a49a64b41b1564f470a0d063cf110ca5aa4ebe WatchSource:0}: Error finding container c26696d8b416e02bf37169f835a49a64b41b1564f470a0d063cf110ca5aa4ebe: Status 404 returned error can't find the container with id c26696d8b416e02bf37169f835a49a64b41b1564f470a0d063cf110ca5aa4ebe Nov 02 22:39:41.276579 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:41.276534 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x"] Nov 02 22:39:41.762065 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:41.762023 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" event={"ID":"f63c140b-171d-4193-b2ec-50ca0e30005d","Type":"ContainerStarted","Data":"6732a4d19c6a3543905470d10952f2d6c8084d175d2d2169b63de3026e5562f3"} Nov 02 22:39:41.762490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:41.762082 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" event={"ID":"f63c140b-171d-4193-b2ec-50ca0e30005d","Type":"ContainerStarted","Data":"c26696d8b416e02bf37169f835a49a64b41b1564f470a0d063cf110ca5aa4ebe"} Nov 02 22:39:41.886930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:41.881903 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" podStartSLOduration=1.8818845560000002 podStartE2EDuration="1.881884556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:41.878380664 +0000 UTC m=+2018.143373883" watchObservedRunningTime="2026-11-02 22:39:41.881884556 +0000 UTC m=+2018.146877779" Nov 02 22:39:42.768280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:42.768245 5109 generic.go:356] "Generic (PLEG): container finished" podID="f63c140b-171d-4193-b2ec-50ca0e30005d" containerID="6732a4d19c6a3543905470d10952f2d6c8084d175d2d2169b63de3026e5562f3" exitCode=0 Nov 02 22:39:42.768752 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:42.768328 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" event={"ID":"f63c140b-171d-4193-b2ec-50ca0e30005d","Type":"ContainerDied","Data":"6732a4d19c6a3543905470d10952f2d6c8084d175d2d2169b63de3026e5562f3"} Nov 02 22:39:43.918293 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:43.918265 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:44.012543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.012506 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-openshift-ca\") pod \"f63c140b-171d-4193-b2ec-50ca0e30005d\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " Nov 02 22:39:44.012770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.012649 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-xks-ca\") pod \"f63c140b-171d-4193-b2ec-50ca0e30005d\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " Nov 02 22:39:44.012849 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.012774 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63c140b-171d-4193-b2ec-50ca0e30005d-kube-api-access-tm5g4\" (UniqueName: \"kubernetes.io/projected/f63c140b-171d-4193-b2ec-50ca0e30005d-kube-api-access-tm5g4\") pod \"f63c140b-171d-4193-b2ec-50ca0e30005d\" (UID: \"f63c140b-171d-4193-b2ec-50ca0e30005d\") " Nov 02 22:39:44.012970 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.012947 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-openshift-ca" pod "f63c140b-171d-4193-b2ec-50ca0e30005d" (UID: "f63c140b-171d-4193-b2ec-50ca0e30005d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:39:44.013105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.013090 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:44.014858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.014828 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-xks-ca" pod "f63c140b-171d-4193-b2ec-50ca0e30005d" (UID: "f63c140b-171d-4193-b2ec-50ca0e30005d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:39:44.015002 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.014947 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63c140b-171d-4193-b2ec-50ca0e30005d-kube-api-access-tm5g4" pod "f63c140b-171d-4193-b2ec-50ca0e30005d" (UID: "f63c140b-171d-4193-b2ec-50ca0e30005d"). InnerVolumeSpecName "kube-api-access-tm5g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:39:44.113716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.113625 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm5g4\" (UniqueName: \"kubernetes.io/projected/f63c140b-171d-4193-b2ec-50ca0e30005d-kube-api-access-tm5g4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:44.113716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.113659 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63c140b-171d-4193-b2ec-50ca0e30005d-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:44.780809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.780769 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" event={"ID":"f63c140b-171d-4193-b2ec-50ca0e30005d","Type":"ContainerDied","Data":"c26696d8b416e02bf37169f835a49a64b41b1564f470a0d063cf110ca5aa4ebe"} Nov 02 22:39:44.781039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.780828 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26696d8b416e02bf37169f835a49a64b41b1564f470a0d063cf110ca5aa4ebe" Nov 02 22:39:44.781039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:44.780780 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x" Nov 02 22:39:46.061604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:46.061551 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg"] Nov 02 22:39:46.112285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:46.112255 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj"] Nov 02 22:39:46.114996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:46.114970 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab95197-9cfwj"] Nov 02 22:39:46.117491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:46.117466 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9bcf4d-2j5jg"] Nov 02 22:39:47.890408 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:47.890372 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d6f956-f6fc-436c-a143-70cb6ed0e2d8" path="/var/lib/kubelet/pods/84d6f956-f6fc-436c-a143-70cb6ed0e2d8/volumes" Nov 02 22:39:47.890875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:39:47.890859 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce75bad1-e1c7-4b05-868f-93c0d209b4af" path="/var/lib/kubelet/pods/ce75bad1-e1c7-4b05-868f-93c0d209b4af/volumes" Nov 02 22:39:56.490549 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:39:56.490506 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:40:11.383525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.383478 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr"] Nov 02 22:40:11.384222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.384144 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f63c140b-171d-4193-b2ec-50ca0e30005d" containerName="revoke-keys" Nov 02 22:40:11.384222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.384166 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63c140b-171d-4193-b2ec-50ca0e30005d" containerName="revoke-keys" Nov 02 22:40:11.384357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.384293 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63c140b-171d-4193-b2ec-50ca0e30005d" containerName="revoke-keys" Nov 02 22:40:11.387402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.387378 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.442936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.442900 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr"] Nov 02 22:40:11.490452 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.490411 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:40:11.490452 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.490443 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:40:11.496998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.496969 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.497073 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.497012 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.497073 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.497049 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfvdh\" (UniqueName: \"kubernetes.io/projected/38ab469b-36fd-4d32-aab4-caa7f33d08e4-kube-api-access-cfvdh\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.598377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.598328 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.598610 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.598415 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.598610 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.598462 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfvdh\" (UniqueName: \"kubernetes.io/projected/38ab469b-36fd-4d32-aab4-caa7f33d08e4-kube-api-access-cfvdh\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.598994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.598975 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.600956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.600938 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.665859 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.665783 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfvdh\" (UniqueName: \"kubernetes.io/projected/38ab469b-36fd-4d32-aab4-caa7f33d08e4-kube-api-access-cfvdh\") pod \"maas-api-revoke-sub-79805ca9-47wjr\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:11.700660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:11.700622 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:12.074121 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:40:12.074090 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ab469b_36fd_4d32_aab4_caa7f33d08e4.slice/crio-f39aaa5883c465672cc57532d68f2cc2398b230f2e34fda8058a68e762eed1f7 WatchSource:0}: Error finding container f39aaa5883c465672cc57532d68f2cc2398b230f2e34fda8058a68e762eed1f7: Status 404 returned error can't find the container with id f39aaa5883c465672cc57532d68f2cc2398b230f2e34fda8058a68e762eed1f7 Nov 02 22:40:12.074350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:12.074324 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr"] Nov 02 22:40:12.943811 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:12.943775 5109 generic.go:356] "Generic (PLEG): container finished" podID="38ab469b-36fd-4d32-aab4-caa7f33d08e4" containerID="2f1da0bff88d2cfa1c84bc391b60e01d040dd4665270cac4ec69afb568751f85" exitCode=0 Nov 02 22:40:12.944275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:12.943867 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" event={"ID":"38ab469b-36fd-4d32-aab4-caa7f33d08e4","Type":"ContainerDied","Data":"2f1da0bff88d2cfa1c84bc391b60e01d040dd4665270cac4ec69afb568751f85"} Nov 02 22:40:12.944275 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:12.943918 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" event={"ID":"38ab469b-36fd-4d32-aab4-caa7f33d08e4","Type":"ContainerStarted","Data":"f39aaa5883c465672cc57532d68f2cc2398b230f2e34fda8058a68e762eed1f7"} Nov 02 22:40:14.085709 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.085677 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:14.226024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.225921 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-xks-ca\") pod \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " Nov 02 22:40:14.226024 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.225994 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ab469b-36fd-4d32-aab4-caa7f33d08e4-kube-api-access-cfvdh\" (UniqueName: \"kubernetes.io/projected/38ab469b-36fd-4d32-aab4-caa7f33d08e4-kube-api-access-cfvdh\") pod \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " Nov 02 22:40:14.226269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.226092 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-openshift-ca\") pod \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\" (UID: \"38ab469b-36fd-4d32-aab4-caa7f33d08e4\") " Nov 02 22:40:14.226553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.226516 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-openshift-ca" pod "38ab469b-36fd-4d32-aab4-caa7f33d08e4" (UID: "38ab469b-36fd-4d32-aab4-caa7f33d08e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:40:14.228357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.228329 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ab469b-36fd-4d32-aab4-caa7f33d08e4-kube-api-access-cfvdh" pod "38ab469b-36fd-4d32-aab4-caa7f33d08e4" (UID: "38ab469b-36fd-4d32-aab4-caa7f33d08e4"). InnerVolumeSpecName "kube-api-access-cfvdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:40:14.228653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.228633 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-xks-ca" pod "38ab469b-36fd-4d32-aab4-caa7f33d08e4" (UID: "38ab469b-36fd-4d32-aab4-caa7f33d08e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:40:14.327267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.327226 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:14.327267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.327260 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfvdh\" (UniqueName: \"kubernetes.io/projected/38ab469b-36fd-4d32-aab4-caa7f33d08e4-kube-api-access-cfvdh\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:14.327267 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.327271 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ab469b-36fd-4d32-aab4-caa7f33d08e4-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:14.955352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.955321 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" Nov 02 22:40:14.955352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.955339 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr" event={"ID":"38ab469b-36fd-4d32-aab4-caa7f33d08e4","Type":"ContainerDied","Data":"f39aaa5883c465672cc57532d68f2cc2398b230f2e34fda8058a68e762eed1f7"} Nov 02 22:40:14.955608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:14.955390 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f39aaa5883c465672cc57532d68f2cc2398b230f2e34fda8058a68e762eed1f7" Nov 02 22:40:17.188034 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:17.188000 5109 scope.go:122] "RemoveContainer" containerID="ab50d7ce7729d45c5b2577cb1adfc4d3396b8985957e7a41f8a80862e364824e" Nov 02 22:40:26.496296 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:40:26.496248 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:40:28.867770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:28.867722 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bdbv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:40:28.868431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:28.868389 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bdbv7" podUID="7dea2654-d13d-484f-ad7a-982fea3bffec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:40:35.239287 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.239239 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5"] Nov 02 22:40:35.239923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.239731 5109 cpu_manager.go:414] "containerMap: removing container" podUID="38ab469b-36fd-4d32-aab4-caa7f33d08e4" containerName="revoke-keys" Nov 02 22:40:35.239923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.239743 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ab469b-36fd-4d32-aab4-caa7f33d08e4" containerName="revoke-keys" Nov 02 22:40:35.239923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.239840 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ab469b-36fd-4d32-aab4-caa7f33d08e4" containerName="revoke-keys" Nov 02 22:40:35.243874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.243848 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.275858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.275810 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.275858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.275857 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2r4p\" (UniqueName: \"kubernetes.io/projected/a791be60-80ac-444c-b200-1ce76fcf0594-kube-api-access-f2r4p\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.276157 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.275937 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.302079 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.302011 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5"] Nov 02 22:40:35.346189 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.346146 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:40:35.346380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.346217 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:40:35.376874 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.376841 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.377027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.376877 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2r4p\" (UniqueName: \"kubernetes.io/projected/a791be60-80ac-444c-b200-1ce76fcf0594-kube-api-access-f2r4p\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.377027 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.376910 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.377501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.377476 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.379431 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.379408 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.440543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.440505 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2r4p\" (UniqueName: \"kubernetes.io/projected/a791be60-80ac-444c-b200-1ce76fcf0594-kube-api-access-f2r4p\") pod \"maas-api-revoke-sub-0b7e8edb-9tsr5\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.558055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.558008 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:35.929960 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:40:35.929914 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda791be60_80ac_444c_b200_1ce76fcf0594.slice/crio-31570ae39f9db95e7460976b4bc02cdc45d301adbb0e5493d4f56848073b925c WatchSource:0}: Error finding container 31570ae39f9db95e7460976b4bc02cdc45d301adbb0e5493d4f56848073b925c: Status 404 returned error can't find the container with id 31570ae39f9db95e7460976b4bc02cdc45d301adbb0e5493d4f56848073b925c Nov 02 22:40:35.931699 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:35.931600 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5"] Nov 02 22:40:36.067983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:36.067939 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" event={"ID":"a791be60-80ac-444c-b200-1ce76fcf0594","Type":"ContainerStarted","Data":"31570ae39f9db95e7460976b4bc02cdc45d301adbb0e5493d4f56848073b925c"} Nov 02 22:40:37.075832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:37.075726 5109 generic.go:356] "Generic (PLEG): container finished" podID="a791be60-80ac-444c-b200-1ce76fcf0594" containerID="80cbbe93e0ed46b5929eac5213133063f3de8e4362234c9725e1172ba703ef87" exitCode=0 Nov 02 22:40:37.075832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:37.075803 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" event={"ID":"a791be60-80ac-444c-b200-1ce76fcf0594","Type":"ContainerDied","Data":"80cbbe93e0ed46b5929eac5213133063f3de8e4362234c9725e1172ba703ef87"} Nov 02 22:40:38.217671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.217645 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:38.306688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.306654 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a791be60-80ac-444c-b200-1ce76fcf0594-kube-api-access-f2r4p\" (UniqueName: \"kubernetes.io/projected/a791be60-80ac-444c-b200-1ce76fcf0594-kube-api-access-f2r4p\") pod \"a791be60-80ac-444c-b200-1ce76fcf0594\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " Nov 02 22:40:38.306877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.306743 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-openshift-ca\") pod \"a791be60-80ac-444c-b200-1ce76fcf0594\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " Nov 02 22:40:38.306877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.306838 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-xks-ca\") pod \"a791be60-80ac-444c-b200-1ce76fcf0594\" (UID: \"a791be60-80ac-444c-b200-1ce76fcf0594\") " Nov 02 22:40:38.307052 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.307033 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-openshift-ca" pod "a791be60-80ac-444c-b200-1ce76fcf0594" (UID: "a791be60-80ac-444c-b200-1ce76fcf0594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:40:38.307165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.307149 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:38.308983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.308949 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-xks-ca" pod "a791be60-80ac-444c-b200-1ce76fcf0594" (UID: "a791be60-80ac-444c-b200-1ce76fcf0594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:40:38.309117 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.309097 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a791be60-80ac-444c-b200-1ce76fcf0594-kube-api-access-f2r4p" pod "a791be60-80ac-444c-b200-1ce76fcf0594" (UID: "a791be60-80ac-444c-b200-1ce76fcf0594"). InnerVolumeSpecName "kube-api-access-f2r4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:40:38.408068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.407960 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2r4p\" (UniqueName: \"kubernetes.io/projected/a791be60-80ac-444c-b200-1ce76fcf0594-kube-api-access-f2r4p\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:38.408068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:38.408012 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a791be60-80ac-444c-b200-1ce76fcf0594-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:39.088215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:39.088186 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" Nov 02 22:40:39.088398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:39.088216 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5" event={"ID":"a791be60-80ac-444c-b200-1ce76fcf0594","Type":"ContainerDied","Data":"31570ae39f9db95e7460976b4bc02cdc45d301adbb0e5493d4f56848073b925c"} Nov 02 22:40:39.088398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:39.088267 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31570ae39f9db95e7460976b4bc02cdc45d301adbb0e5493d4f56848073b925c" Nov 02 22:40:46.694668 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.694555 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt"] Nov 02 22:40:46.695280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.695210 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a791be60-80ac-444c-b200-1ce76fcf0594" containerName="revoke-keys" Nov 02 22:40:46.695280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.695231 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a791be60-80ac-444c-b200-1ce76fcf0594" containerName="revoke-keys" Nov 02 22:40:46.695409 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.695356 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a791be60-80ac-444c-b200-1ce76fcf0594" containerName="revoke-keys" Nov 02 22:40:46.698791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.698769 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.754136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.754101 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt"] Nov 02 22:40:46.787995 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.787957 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.788171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.788012 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr6h\" (UniqueName: \"kubernetes.io/projected/30ddb749-c79e-4149-a78a-c10a46ec789b-kube-api-access-prr6h\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.788239 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.788168 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.801179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.801140 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:40:46.801179 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.801161 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:40:46.889541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.889504 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.889765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.889593 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.889765 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.889733 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr6h\" (UniqueName: \"kubernetes.io/projected/30ddb749-c79e-4149-a78a-c10a46ec789b-kube-api-access-prr6h\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.890280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.890259 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.892091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.892073 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:46.952535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:46.952445 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr6h\" (UniqueName: \"kubernetes.io/projected/30ddb749-c79e-4149-a78a-c10a46ec789b-kube-api-access-prr6h\") pod \"maas-api-revoke-sub-3d4156f1-g45lt\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:47.011916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:47.011877 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:47.389477 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:40:47.389436 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ddb749_c79e_4149_a78a_c10a46ec789b.slice/crio-717209bd2264b81b7f9e910dff54edb6ef7d5df6890578f73adb9e53103fb9d5 WatchSource:0}: Error finding container 717209bd2264b81b7f9e910dff54edb6ef7d5df6890578f73adb9e53103fb9d5: Status 404 returned error can't find the container with id 717209bd2264b81b7f9e910dff54edb6ef7d5df6890578f73adb9e53103fb9d5 Nov 02 22:40:47.390401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:47.390377 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt"] Nov 02 22:40:48.057938 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:48.057902 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2"] Nov 02 22:40:48.060745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:48.060708 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9jpm2"] Nov 02 22:40:48.135618 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:48.135577 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" event={"ID":"30ddb749-c79e-4149-a78a-c10a46ec789b","Type":"ContainerStarted","Data":"e2965f5621af5c8a470862f3a967eda72a2addad888df1271bdff709d28fb241"} Nov 02 22:40:48.135836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:48.135635 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" event={"ID":"30ddb749-c79e-4149-a78a-c10a46ec789b","Type":"ContainerStarted","Data":"717209bd2264b81b7f9e910dff54edb6ef7d5df6890578f73adb9e53103fb9d5"} Nov 02 22:40:48.254450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:48.254389 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" podStartSLOduration=2.2543755819999998 podStartE2EDuration="2.254375582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:40:48.252485209 +0000 UTC m=+2084.517478428" watchObservedRunningTime="2026-11-02 22:40:48.254375582 +0000 UTC m=+2084.519368844" Nov 02 22:40:49.154820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:49.154780 5109 generic.go:356] "Generic (PLEG): container finished" podID="30ddb749-c79e-4149-a78a-c10a46ec789b" containerID="e2965f5621af5c8a470862f3a967eda72a2addad888df1271bdff709d28fb241" exitCode=0 Nov 02 22:40:49.165630 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:49.165587 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" event={"ID":"30ddb749-c79e-4149-a78a-c10a46ec789b","Type":"ContainerDied","Data":"e2965f5621af5c8a470862f3a967eda72a2addad888df1271bdff709d28fb241"} Nov 02 22:40:49.897076 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:49.897040 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a65cc06-3b3a-40a3-b8c2-0fdd056cd592" path="/var/lib/kubelet/pods/4a65cc06-3b3a-40a3-b8c2-0fdd056cd592/volumes" Nov 02 22:40:49.998705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:49.998670 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jkwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:40:49.998957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:49.998935 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jkwk" podUID="c7eafe82-f30e-429e-907e-0d60710314a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:40:50.305258 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.305233 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:50.434039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.433963 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-openshift-ca\") pod \"30ddb749-c79e-4149-a78a-c10a46ec789b\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " Nov 02 22:40:50.434039 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.434030 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ddb749-c79e-4149-a78a-c10a46ec789b-kube-api-access-prr6h\" (UniqueName: \"kubernetes.io/projected/30ddb749-c79e-4149-a78a-c10a46ec789b-kube-api-access-prr6h\") pod \"30ddb749-c79e-4149-a78a-c10a46ec789b\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " Nov 02 22:40:50.434254 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.434067 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-xks-ca\") pod \"30ddb749-c79e-4149-a78a-c10a46ec789b\" (UID: \"30ddb749-c79e-4149-a78a-c10a46ec789b\") " Nov 02 22:40:50.434421 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.434395 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-openshift-ca" pod "30ddb749-c79e-4149-a78a-c10a46ec789b" (UID: "30ddb749-c79e-4149-a78a-c10a46ec789b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:40:50.436485 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.436451 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-xks-ca" pod "30ddb749-c79e-4149-a78a-c10a46ec789b" (UID: "30ddb749-c79e-4149-a78a-c10a46ec789b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:40:50.436601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.436520 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ddb749-c79e-4149-a78a-c10a46ec789b-kube-api-access-prr6h" pod "30ddb749-c79e-4149-a78a-c10a46ec789b" (UID: "30ddb749-c79e-4149-a78a-c10a46ec789b"). InnerVolumeSpecName "kube-api-access-prr6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:40:50.535045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.535004 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:50.535045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.535039 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prr6h\" (UniqueName: \"kubernetes.io/projected/30ddb749-c79e-4149-a78a-c10a46ec789b-kube-api-access-prr6h\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:50.535045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:50.535050 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ddb749-c79e-4149-a78a-c10a46ec789b-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:51.168167 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:51.168127 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" event={"ID":"30ddb749-c79e-4149-a78a-c10a46ec789b","Type":"ContainerDied","Data":"717209bd2264b81b7f9e910dff54edb6ef7d5df6890578f73adb9e53103fb9d5"} Nov 02 22:40:51.168548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:51.168142 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt" Nov 02 22:40:51.168548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:40:51.168217 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="717209bd2264b81b7f9e910dff54edb6ef7d5df6890578f73adb9e53103fb9d5" Nov 02 22:40:56.490696 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:40:56.490656 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:41:17.213760 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.213728 5109 scope.go:122] "RemoveContainer" containerID="767fc7efd35b20285917d199c96a880e47cd01d7077253c4e28792ecf909a3cd" Nov 02 22:41:17.223240 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.223218 5109 scope.go:122] "RemoveContainer" containerID="2a739c320d33e2f3851d5047c86ddc73487e4d8d50011ed04ff6b8f11500fe80" Nov 02 22:41:17.231749 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.231726 5109 scope.go:122] "RemoveContainer" containerID="be415ae9db953fdbcb2dac2d80d9cdbb58367d7f007329123efe175d1c2bd8a3" Nov 02 22:41:17.314217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.314185 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt"] Nov 02 22:41:17.364098 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.364065 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g45lt"] Nov 02 22:41:17.433118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.433083 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf"] Nov 02 22:41:17.433540 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.433526 5109 cpu_manager.go:414] "containerMap: removing container" podUID="30ddb749-c79e-4149-a78a-c10a46ec789b" containerName="revoke-keys" Nov 02 22:41:17.433619 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.433542 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ddb749-c79e-4149-a78a-c10a46ec789b" containerName="revoke-keys" Nov 02 22:41:17.433705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.433693 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ddb749-c79e-4149-a78a-c10a46ec789b" containerName="revoke-keys" Nov 02 22:41:17.438484 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.438461 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.498649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.496599 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:41:17.498649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.496611 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:41:17.498649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.498639 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf"] Nov 02 22:41:17.568234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.568186 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4778\" (UniqueName: \"kubernetes.io/projected/bb01d13a-13b7-438e-8b21-5084f0d8599c-kube-api-access-x4778\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.568456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.568394 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.568456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.568434 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.669645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.669610 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.669645 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.669652 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.669973 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.669701 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4778\" (UniqueName: \"kubernetes.io/projected/bb01d13a-13b7-438e-8b21-5084f0d8599c-kube-api-access-x4778\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.670280 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.670249 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.672211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.672190 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.731233 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.731203 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4778\" (UniqueName: \"kubernetes.io/projected/bb01d13a-13b7-438e-8b21-5084f0d8599c-kube-api-access-x4778\") pod \"maas-api-revoke-sub-3d4156f1-gbrvf\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.754270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.754185 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:17.890521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:17.890482 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ddb749-c79e-4149-a78a-c10a46ec789b" path="/var/lib/kubelet/pods/30ddb749-c79e-4149-a78a-c10a46ec789b/volumes" Nov 02 22:41:18.125377 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:41:18.125343 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb01d13a_13b7_438e_8b21_5084f0d8599c.slice/crio-a6e04e32ee9d86272b220fc694f31d9d38a7bd3cbdf162f913bc3938db1e70e1 WatchSource:0}: Error finding container a6e04e32ee9d86272b220fc694f31d9d38a7bd3cbdf162f913bc3938db1e70e1: Status 404 returned error can't find the container with id a6e04e32ee9d86272b220fc694f31d9d38a7bd3cbdf162f913bc3938db1e70e1 Nov 02 22:41:18.126838 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:18.126815 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf"] Nov 02 22:41:18.354914 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:18.354882 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" event={"ID":"bb01d13a-13b7-438e-8b21-5084f0d8599c","Type":"ContainerStarted","Data":"a6e04e32ee9d86272b220fc694f31d9d38a7bd3cbdf162f913bc3938db1e70e1"} Nov 02 22:41:19.361580 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:19.361527 5109 generic.go:356] "Generic (PLEG): container finished" podID="bb01d13a-13b7-438e-8b21-5084f0d8599c" containerID="be0945ac34a901df9d8390a446299a576c8e415cd0340b6dc81319be37830785" exitCode=0 Nov 02 22:41:19.361977 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:19.361607 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" event={"ID":"bb01d13a-13b7-438e-8b21-5084f0d8599c","Type":"ContainerDied","Data":"be0945ac34a901df9d8390a446299a576c8e415cd0340b6dc81319be37830785"} Nov 02 22:41:20.506120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.506090 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:20.597270 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.597226 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb01d13a-13b7-438e-8b21-5084f0d8599c-kube-api-access-x4778\" (UniqueName: \"kubernetes.io/projected/bb01d13a-13b7-438e-8b21-5084f0d8599c-kube-api-access-x4778\") pod \"bb01d13a-13b7-438e-8b21-5084f0d8599c\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " Nov 02 22:41:20.597435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.597354 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-xks-ca\") pod \"bb01d13a-13b7-438e-8b21-5084f0d8599c\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " Nov 02 22:41:20.597435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.597399 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-openshift-ca\") pod \"bb01d13a-13b7-438e-8b21-5084f0d8599c\" (UID: \"bb01d13a-13b7-438e-8b21-5084f0d8599c\") " Nov 02 22:41:20.597858 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.597833 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-openshift-ca" pod "bb01d13a-13b7-438e-8b21-5084f0d8599c" (UID: "bb01d13a-13b7-438e-8b21-5084f0d8599c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:41:20.599518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.599495 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb01d13a-13b7-438e-8b21-5084f0d8599c-kube-api-access-x4778" pod "bb01d13a-13b7-438e-8b21-5084f0d8599c" (UID: "bb01d13a-13b7-438e-8b21-5084f0d8599c"). InnerVolumeSpecName "kube-api-access-x4778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:41:20.599643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.599560 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-xks-ca" pod "bb01d13a-13b7-438e-8b21-5084f0d8599c" (UID: "bb01d13a-13b7-438e-8b21-5084f0d8599c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:20.698601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.698473 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4778\" (UniqueName: \"kubernetes.io/projected/bb01d13a-13b7-438e-8b21-5084f0d8599c-kube-api-access-x4778\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:20.698601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.698510 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:20.698601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:20.698526 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb01d13a-13b7-438e-8b21-5084f0d8599c-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:21.374460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:21.374422 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" Nov 02 22:41:21.374660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:21.374461 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf" event={"ID":"bb01d13a-13b7-438e-8b21-5084f0d8599c","Type":"ContainerDied","Data":"a6e04e32ee9d86272b220fc694f31d9d38a7bd3cbdf162f913bc3938db1e70e1"} Nov 02 22:41:21.374660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:21.374517 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e04e32ee9d86272b220fc694f31d9d38a7bd3cbdf162f913bc3938db1e70e1" Nov 02 22:41:26.491445 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:41:26.491406 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:41:56.489927 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:41:56.489883 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:41:58.100716 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:58.100638 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb"] Nov 02 22:41:58.101948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:58.101717 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kzjqb"] Nov 02 22:41:59.899609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:41:59.899181 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88f8ead-e8d2-4191-82db-355962649063" path="/var/lib/kubelet/pods/d88f8ead-e8d2-4191-82db-355962649063/volumes" Nov 02 22:42:17.263898 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:17.263815 5109 scope.go:122] "RemoveContainer" containerID="727a221e5cbf20f1be4b671d197409d6d660a7fa2e7800892b9b1922b878bfd0" Nov 02 22:42:17.927226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:17.927180 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv"] Nov 02 22:42:17.927767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:17.927747 5109 cpu_manager.go:414] "containerMap: removing container" podUID="bb01d13a-13b7-438e-8b21-5084f0d8599c" containerName="revoke-keys" Nov 02 22:42:17.927893 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:17.927767 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb01d13a-13b7-438e-8b21-5084f0d8599c" containerName="revoke-keys" Nov 02 22:42:17.927956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:17.927894 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb01d13a-13b7-438e-8b21-5084f0d8599c" containerName="revoke-keys" Nov 02 22:42:17.932435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:17.932412 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:17.990035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:17.989999 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv"] Nov 02 22:42:18.037201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.037159 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:42:18.037385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.037268 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:42:18.086265 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.086218 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.086465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.086332 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.086465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.086352 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4cnw\" (UniqueName: \"kubernetes.io/projected/7301b52b-2168-45b6-b6a6-6a5ff794b1af-kube-api-access-d4cnw\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.187597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.187481 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.187597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.187515 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4cnw\" (UniqueName: \"kubernetes.io/projected/7301b52b-2168-45b6-b6a6-6a5ff794b1af-kube-api-access-d4cnw\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.187861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.187620 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.188262 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.188235 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.190175 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.190152 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.249553 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.249512 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4cnw\" (UniqueName: \"kubernetes.io/projected/7301b52b-2168-45b6-b6a6-6a5ff794b1af-kube-api-access-d4cnw\") pod \"maas-api-revoke-sub-e2fc9c5b-g4wxv\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.545415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.545370 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:18.914600 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:42:18.914539 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7301b52b_2168_45b6_b6a6_6a5ff794b1af.slice/crio-865cb0a240f91f0009c88e368d0b8a73e99006b3e32b8a374b24dabd281c5f7a WatchSource:0}: Error finding container 865cb0a240f91f0009c88e368d0b8a73e99006b3e32b8a374b24dabd281c5f7a: Status 404 returned error can't find the container with id 865cb0a240f91f0009c88e368d0b8a73e99006b3e32b8a374b24dabd281c5f7a Nov 02 22:42:18.916271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:18.916115 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv"] Nov 02 22:42:19.696770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:19.696726 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" event={"ID":"7301b52b-2168-45b6-b6a6-6a5ff794b1af","Type":"ContainerStarted","Data":"f52056478f03a503cabbd58b035ceb650e3e4173002a382ce7adf417fe013a2a"} Nov 02 22:42:19.697272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:19.696787 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" event={"ID":"7301b52b-2168-45b6-b6a6-6a5ff794b1af","Type":"ContainerStarted","Data":"865cb0a240f91f0009c88e368d0b8a73e99006b3e32b8a374b24dabd281c5f7a"} Nov 02 22:42:19.813870 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:19.813815 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" podStartSLOduration=2.813800567 podStartE2EDuration="2.813800567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:42:19.812184837 +0000 UTC m=+2176.077178053" watchObservedRunningTime="2026-11-02 22:42:19.813800567 +0000 UTC m=+2176.078793836" Nov 02 22:42:20.704933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:20.704896 5109 generic.go:356] "Generic (PLEG): container finished" podID="7301b52b-2168-45b6-b6a6-6a5ff794b1af" containerID="f52056478f03a503cabbd58b035ceb650e3e4173002a382ce7adf417fe013a2a" exitCode=0 Nov 02 22:42:20.704933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:20.704941 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" event={"ID":"7301b52b-2168-45b6-b6a6-6a5ff794b1af","Type":"ContainerDied","Data":"f52056478f03a503cabbd58b035ceb650e3e4173002a382ce7adf417fe013a2a"} Nov 02 22:42:21.840971 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:21.840940 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:21.924879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:21.924841 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-xks-ca\") pod \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " Nov 02 22:42:21.925101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:21.924931 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-openshift-ca\") pod \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " Nov 02 22:42:21.925101 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:21.924991 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7301b52b-2168-45b6-b6a6-6a5ff794b1af-kube-api-access-d4cnw\" (UniqueName: \"kubernetes.io/projected/7301b52b-2168-45b6-b6a6-6a5ff794b1af-kube-api-access-d4cnw\") pod \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\" (UID: \"7301b52b-2168-45b6-b6a6-6a5ff794b1af\") " Nov 02 22:42:21.925351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:21.925327 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-openshift-ca" pod "7301b52b-2168-45b6-b6a6-6a5ff794b1af" (UID: "7301b52b-2168-45b6-b6a6-6a5ff794b1af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:42:21.927106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:21.927082 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-xks-ca" pod "7301b52b-2168-45b6-b6a6-6a5ff794b1af" (UID: "7301b52b-2168-45b6-b6a6-6a5ff794b1af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:42:21.927210 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:21.927129 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7301b52b-2168-45b6-b6a6-6a5ff794b1af-kube-api-access-d4cnw" pod "7301b52b-2168-45b6-b6a6-6a5ff794b1af" (UID: "7301b52b-2168-45b6-b6a6-6a5ff794b1af"). InnerVolumeSpecName "kube-api-access-d4cnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:22.026402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:22.026365 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:22.026402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:22.026397 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4cnw\" (UniqueName: \"kubernetes.io/projected/7301b52b-2168-45b6-b6a6-6a5ff794b1af-kube-api-access-d4cnw\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:22.026402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:22.026409 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7301b52b-2168-45b6-b6a6-6a5ff794b1af-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:22.717006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:22.716973 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" Nov 02 22:42:22.717197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:22.716972 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv" event={"ID":"7301b52b-2168-45b6-b6a6-6a5ff794b1af","Type":"ContainerDied","Data":"865cb0a240f91f0009c88e368d0b8a73e99006b3e32b8a374b24dabd281c5f7a"} Nov 02 22:42:22.717197 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:22.717131 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="865cb0a240f91f0009c88e368d0b8a73e99006b3e32b8a374b24dabd281c5f7a" Nov 02 22:42:26.490424 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:42:26.490384 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:42:28.752432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.749156 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv"] Nov 02 22:42:28.752432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.750123 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7301b52b-2168-45b6-b6a6-6a5ff794b1af" containerName="revoke-keys" Nov 02 22:42:28.752432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.750150 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7301b52b-2168-45b6-b6a6-6a5ff794b1af" containerName="revoke-keys" Nov 02 22:42:28.752432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.750382 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="7301b52b-2168-45b6-b6a6-6a5ff794b1af" containerName="revoke-keys" Nov 02 22:42:28.756335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.756301 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.806285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.806251 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv"] Nov 02 22:42:28.853670 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.853635 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:42:28.853861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.853685 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:42:28.894629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.894550 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.894831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.894693 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79ms\" (UniqueName: \"kubernetes.io/projected/b26b829b-66af-49d1-a9f4-4d20285ddb08-kube-api-access-v79ms\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.894831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.894774 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.995900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.995850 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.996120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.995908 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79ms\" (UniqueName: \"kubernetes.io/projected/b26b829b-66af-49d1-a9f4-4d20285ddb08-kube-api-access-v79ms\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.996120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.995965 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.996700 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.996679 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:28.998435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:28.998416 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:29.057747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:29.057712 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79ms\" (UniqueName: \"kubernetes.io/projected/b26b829b-66af-49d1-a9f4-4d20285ddb08-kube-api-access-v79ms\") pod \"maas-api-revoke-sub-5a29efb3-q42qv\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:29.069905 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:29.069874 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:29.439053 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:42:29.439005 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26b829b_66af_49d1_a9f4_4d20285ddb08.slice/crio-1c1387a1b61d3f8b15296c2e98e6da96f2fbe79320ca70058afc3d6f6b6a2766 WatchSource:0}: Error finding container 1c1387a1b61d3f8b15296c2e98e6da96f2fbe79320ca70058afc3d6f6b6a2766: Status 404 returned error can't find the container with id 1c1387a1b61d3f8b15296c2e98e6da96f2fbe79320ca70058afc3d6f6b6a2766 Nov 02 22:42:29.440612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:29.440583 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv"] Nov 02 22:42:29.764292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:29.764249 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" event={"ID":"b26b829b-66af-49d1-a9f4-4d20285ddb08","Type":"ContainerStarted","Data":"2af07a22d58b91361575b3924ec6d51903e35307c32e2f8a535ec5ea33623454"} Nov 02 22:42:29.764750 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:29.764324 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" event={"ID":"b26b829b-66af-49d1-a9f4-4d20285ddb08","Type":"ContainerStarted","Data":"1c1387a1b61d3f8b15296c2e98e6da96f2fbe79320ca70058afc3d6f6b6a2766"} Nov 02 22:42:29.882918 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:29.882852 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" podStartSLOduration=1.882836419 podStartE2EDuration="1.882836419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:42:29.88135994 +0000 UTC m=+2186.146353160" watchObservedRunningTime="2026-11-02 22:42:29.882836419 +0000 UTC m=+2186.147829637" Nov 02 22:42:30.048986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:30.048948 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp"] Nov 02 22:42:30.052913 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:30.052886 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mn4kp"] Nov 02 22:42:30.771314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:30.771279 5109 generic.go:356] "Generic (PLEG): container finished" podID="b26b829b-66af-49d1-a9f4-4d20285ddb08" containerID="2af07a22d58b91361575b3924ec6d51903e35307c32e2f8a535ec5ea33623454" exitCode=0 Nov 02 22:42:30.771732 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:30.771359 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" event={"ID":"b26b829b-66af-49d1-a9f4-4d20285ddb08","Type":"ContainerDied","Data":"2af07a22d58b91361575b3924ec6d51903e35307c32e2f8a535ec5ea33623454"} Nov 02 22:42:31.894719 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:31.894677 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3352cfbd-c5c7-4686-83c9-eb9905b51d38" path="/var/lib/kubelet/pods/3352cfbd-c5c7-4686-83c9-eb9905b51d38/volumes" Nov 02 22:42:31.948448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:31.948413 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:32.143667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.142166 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-xks-ca\") pod \"b26b829b-66af-49d1-a9f4-4d20285ddb08\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " Nov 02 22:42:32.143667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.142249 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-openshift-ca\") pod \"b26b829b-66af-49d1-a9f4-4d20285ddb08\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " Nov 02 22:42:32.143667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.142282 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26b829b-66af-49d1-a9f4-4d20285ddb08-kube-api-access-v79ms\" (UniqueName: \"kubernetes.io/projected/b26b829b-66af-49d1-a9f4-4d20285ddb08-kube-api-access-v79ms\") pod \"b26b829b-66af-49d1-a9f4-4d20285ddb08\" (UID: \"b26b829b-66af-49d1-a9f4-4d20285ddb08\") " Nov 02 22:42:32.147605 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.147282 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-openshift-ca" pod "b26b829b-66af-49d1-a9f4-4d20285ddb08" (UID: "b26b829b-66af-49d1-a9f4-4d20285ddb08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:42:32.176802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.176756 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26b829b-66af-49d1-a9f4-4d20285ddb08-kube-api-access-v79ms" pod "b26b829b-66af-49d1-a9f4-4d20285ddb08" (UID: "b26b829b-66af-49d1-a9f4-4d20285ddb08"). InnerVolumeSpecName "kube-api-access-v79ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:32.177129 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.177072 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-xks-ca" pod "b26b829b-66af-49d1-a9f4-4d20285ddb08" (UID: "b26b829b-66af-49d1-a9f4-4d20285ddb08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:42:32.247301 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.247263 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79ms\" (UniqueName: \"kubernetes.io/projected/b26b829b-66af-49d1-a9f4-4d20285ddb08-kube-api-access-v79ms\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:32.247551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.247536 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:32.247680 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.247668 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26b829b-66af-49d1-a9f4-4d20285ddb08-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:32.795003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.794964 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" Nov 02 22:42:32.795277 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.794971 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv" event={"ID":"b26b829b-66af-49d1-a9f4-4d20285ddb08","Type":"ContainerDied","Data":"1c1387a1b61d3f8b15296c2e98e6da96f2fbe79320ca70058afc3d6f6b6a2766"} Nov 02 22:42:32.795492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:32.795476 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1387a1b61d3f8b15296c2e98e6da96f2fbe79320ca70058afc3d6f6b6a2766" Nov 02 22:42:52.051215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:52.051173 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx"] Nov 02 22:42:52.101084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:52.101045 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xzqjx"] Nov 02 22:42:53.890698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:53.890665 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ae9319-0ed0-44b3-b252-7cdce332b086" path="/var/lib/kubelet/pods/13ae9319-0ed0-44b3-b252-7cdce332b086/volumes" Nov 02 22:42:56.080058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.080011 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf"] Nov 02 22:42:56.130084 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.130038 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbrvf"] Nov 02 22:42:56.208170 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.208127 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k"] Nov 02 22:42:56.208825 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.208808 5109 cpu_manager.go:414] "containerMap: removing container" podUID="b26b829b-66af-49d1-a9f4-4d20285ddb08" containerName="revoke-keys" Nov 02 22:42:56.208889 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.208828 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26b829b-66af-49d1-a9f4-4d20285ddb08" containerName="revoke-keys" Nov 02 22:42:56.208941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.208928 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26b829b-66af-49d1-a9f4-4d20285ddb08" containerName="revoke-keys" Nov 02 22:42:56.215474 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.215438 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.267695 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.267659 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k"] Nov 02 22:42:56.268495 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.268466 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:42:56.268706 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.268601 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:42:56.305171 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.305129 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwphs\" (UniqueName: \"kubernetes.io/projected/c1f3a192-db58-4e00-8e61-2290273186c4-kube-api-access-vwphs\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.305360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.305319 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.305410 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.305372 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.406542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.406451 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwphs\" (UniqueName: \"kubernetes.io/projected/c1f3a192-db58-4e00-8e61-2290273186c4-kube-api-access-vwphs\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.406542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.406536 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.406779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.406593 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.407356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.407290 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.409184 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.409154 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.468187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.468143 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwphs\" (UniqueName: \"kubernetes.io/projected/c1f3a192-db58-4e00-8e61-2290273186c4-kube-api-access-vwphs\") pod \"maas-api-revoke-sub-3d4156f1-p2k8k\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.491073 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:42:56.491033 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:42:56.529464 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.529427 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:56.899487 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:42:56.899455 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f3a192_db58_4e00_8e61_2290273186c4.slice/crio-04df55c052300ea23634cb057b6b3c4dc4e7c5e424e2c3eaefc293ba46f05c13 WatchSource:0}: Error finding container 04df55c052300ea23634cb057b6b3c4dc4e7c5e424e2c3eaefc293ba46f05c13: Status 404 returned error can't find the container with id 04df55c052300ea23634cb057b6b3c4dc4e7c5e424e2c3eaefc293ba46f05c13 Nov 02 22:42:56.901325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.901289 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k"] Nov 02 22:42:56.922268 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:56.922239 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" event={"ID":"c1f3a192-db58-4e00-8e61-2290273186c4","Type":"ContainerStarted","Data":"04df55c052300ea23634cb057b6b3c4dc4e7c5e424e2c3eaefc293ba46f05c13"} Nov 02 22:42:57.891040 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:57.891007 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb01d13a-13b7-438e-8b21-5084f0d8599c" path="/var/lib/kubelet/pods/bb01d13a-13b7-438e-8b21-5084f0d8599c/volumes" Nov 02 22:42:57.928892 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:57.928855 5109 generic.go:356] "Generic (PLEG): container finished" podID="c1f3a192-db58-4e00-8e61-2290273186c4" containerID="779d87ff6b4abef256dc099a2ea5d81a6f89b54f1f466ba7c2ba4ed9e61d5a2c" exitCode=0 Nov 02 22:42:57.929075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:57.928911 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" event={"ID":"c1f3a192-db58-4e00-8e61-2290273186c4","Type":"ContainerDied","Data":"779d87ff6b4abef256dc099a2ea5d81a6f89b54f1f466ba7c2ba4ed9e61d5a2c"} Nov 02 22:42:59.082164 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.082135 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:59.234910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.234807 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-openshift-ca\") pod \"c1f3a192-db58-4e00-8e61-2290273186c4\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " Nov 02 22:42:59.234910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.234865 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f3a192-db58-4e00-8e61-2290273186c4-kube-api-access-vwphs\" (UniqueName: \"kubernetes.io/projected/c1f3a192-db58-4e00-8e61-2290273186c4-kube-api-access-vwphs\") pod \"c1f3a192-db58-4e00-8e61-2290273186c4\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " Nov 02 22:42:59.234910 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.234892 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-xks-ca\") pod \"c1f3a192-db58-4e00-8e61-2290273186c4\" (UID: \"c1f3a192-db58-4e00-8e61-2290273186c4\") " Nov 02 22:42:59.235279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.235253 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-openshift-ca" pod "c1f3a192-db58-4e00-8e61-2290273186c4" (UID: "c1f3a192-db58-4e00-8e61-2290273186c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:42:59.237305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.237266 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-xks-ca" pod "c1f3a192-db58-4e00-8e61-2290273186c4" (UID: "c1f3a192-db58-4e00-8e61-2290273186c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:42:59.237305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.237272 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f3a192-db58-4e00-8e61-2290273186c4-kube-api-access-vwphs" pod "c1f3a192-db58-4e00-8e61-2290273186c4" (UID: "c1f3a192-db58-4e00-8e61-2290273186c4"). InnerVolumeSpecName "kube-api-access-vwphs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:59.335891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.335844 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:59.335891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.335880 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwphs\" (UniqueName: \"kubernetes.io/projected/c1f3a192-db58-4e00-8e61-2290273186c4-kube-api-access-vwphs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:59.335891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.335891 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3a192-db58-4e00-8e61-2290273186c4-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:59.941199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.941169 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" Nov 02 22:42:59.941199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.941177 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k" event={"ID":"c1f3a192-db58-4e00-8e61-2290273186c4","Type":"ContainerDied","Data":"04df55c052300ea23634cb057b6b3c4dc4e7c5e424e2c3eaefc293ba46f05c13"} Nov 02 22:42:59.941466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:42:59.941226 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04df55c052300ea23634cb057b6b3c4dc4e7c5e424e2c3eaefc293ba46f05c13" Nov 02 22:43:12.459734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:12.459690 5109 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tk2q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:43:12.460423 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:12.460386 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tk2q6" podUID="c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:43:17.286125 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:17.286085 5109 scope.go:122] "RemoveContainer" containerID="f9e713b8b090ddf51f1b501628f57937081bea8a721f4d862d413001afba0048" Nov 02 22:43:26.490096 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:43:26.490054 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:43:27.050921 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:27.050880 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks"] Nov 02 22:43:27.102118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:27.102075 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c5vks"] Nov 02 22:43:27.890744 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:27.890704 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727ee4e1-f320-4ae3-b8eb-d085936af589" path="/var/lib/kubelet/pods/727ee4e1-f320-4ae3-b8eb-d085936af589/volumes" Nov 02 22:43:38.051632 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:38.051587 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz"] Nov 02 22:43:38.101356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:38.101320 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k4mtz"] Nov 02 22:43:39.890672 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:39.890635 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d1ec81-1928-4772-9204-79e9cd939434" path="/var/lib/kubelet/pods/c6d1ec81-1928-4772-9204-79e9cd939434/volumes" Nov 02 22:43:46.643638 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.643511 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm"] Nov 02 22:43:46.644042 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.643997 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c1f3a192-db58-4e00-8e61-2290273186c4" containerName="revoke-keys" Nov 02 22:43:46.644042 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.644009 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f3a192-db58-4e00-8e61-2290273186c4" containerName="revoke-keys" Nov 02 22:43:46.644119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.644080 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f3a192-db58-4e00-8e61-2290273186c4" containerName="revoke-keys" Nov 02 22:43:46.648329 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.648313 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:46.738405 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.738366 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50e1106-768b-4ba5-9961-d22df26b8369-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:46.738624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.738433 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9zn\" (UniqueName: \"kubernetes.io/projected/e50e1106-768b-4ba5-9961-d22df26b8369-kube-api-access-mh9zn\") pod \"payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:46.756404 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.756364 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a8baae\"" Nov 02 22:43:46.756655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.756453 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-22bcf6df-dockercfg-l8dts\"" Nov 02 22:43:46.774272 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.774230 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm"] Nov 02 22:43:46.808460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.808423 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj"] Nov 02 22:43:46.812069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.812047 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:46.839703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.839673 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9zn\" (UniqueName: \"kubernetes.io/projected/e50e1106-768b-4ba5-9961-d22df26b8369-kube-api-access-mh9zn\") pod \"payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:46.839867 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.839775 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50e1106-768b-4ba5-9961-d22df26b8369-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:46.840441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.840419 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50e1106-768b-4ba5-9961-d22df26b8369-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:46.860260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.860228 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj"] Nov 02 22:43:46.901035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.900968 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9zn\" (UniqueName: \"kubernetes.io/projected/e50e1106-768b-4ba5-9961-d22df26b8369-kube-api-access-mh9zn\") pod \"payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:46.940819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.940781 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8rt\" (UniqueName: \"kubernetes.io/projected/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-kube-api-access-4b8rt\") pod \"payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:46.940819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.940817 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:46.961257 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:46.961221 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:47.041368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:47.041330 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8rt\" (UniqueName: \"kubernetes.io/projected/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-kube-api-access-4b8rt\") pod \"payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:47.041592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:47.041372 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:47.042166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:47.042145 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:47.102769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:47.102729 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8rt\" (UniqueName: \"kubernetes.io/projected/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-kube-api-access-4b8rt\") pod \"payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:47.125486 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:47.125457 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:47.333282 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:43:47.333248 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50e1106_768b_4ba5_9961_d22df26b8369.slice/crio-0c7d3edf50c8fdb131e136d14d84bb6c56ca300d21545d5da6a68451bc49236d WatchSource:0}: Error finding container 0c7d3edf50c8fdb131e136d14d84bb6c56ca300d21545d5da6a68451bc49236d: Status 404 returned error can't find the container with id 0c7d3edf50c8fdb131e136d14d84bb6c56ca300d21545d5da6a68451bc49236d Nov 02 22:43:47.334799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:47.334780 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm"] Nov 02 22:43:47.497516 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:43:47.497478 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538dbdd7_c60c_4dd7_9b2a_8877f0d3537f.slice/crio-1bfa758713eb9e6a0ce823277362a0547a0cb5a3f4d8dc55e2de4b6183a76bb9 WatchSource:0}: Error finding container 1bfa758713eb9e6a0ce823277362a0547a0cb5a3f4d8dc55e2de4b6183a76bb9: Status 404 returned error can't find the container with id 1bfa758713eb9e6a0ce823277362a0547a0cb5a3f4d8dc55e2de4b6183a76bb9 Nov 02 22:43:47.498956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:47.498932 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj"] Nov 02 22:43:48.219454 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.219412 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" event={"ID":"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f","Type":"ContainerStarted","Data":"70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386"} Nov 02 22:43:48.220003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.219487 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" event={"ID":"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f","Type":"ContainerStarted","Data":"1bfa758713eb9e6a0ce823277362a0547a0cb5a3f4d8dc55e2de4b6183a76bb9"} Nov 02 22:43:48.220003 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.219539 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:48.221054 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.221029 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" event={"ID":"e50e1106-768b-4ba5-9961-d22df26b8369","Type":"ContainerStarted","Data":"729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3"} Nov 02 22:43:48.221181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.221069 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" event={"ID":"e50e1106-768b-4ba5-9961-d22df26b8369","Type":"ContainerStarted","Data":"0c7d3edf50c8fdb131e136d14d84bb6c56ca300d21545d5da6a68451bc49236d"} Nov 02 22:43:48.221181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.221099 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:43:48.338314 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.335434 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" podStartSLOduration=2.33541552 podStartE2EDuration="2.33541552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:48.335132947 +0000 UTC m=+2264.600126167" watchObservedRunningTime="2026-11-02 22:43:48.33541552 +0000 UTC m=+2264.600408740" Nov 02 22:43:48.456061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:48.455018 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" podStartSLOduration=2.454997936 podStartE2EDuration="2.454997936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:48.453735381 +0000 UTC m=+2264.718728601" watchObservedRunningTime="2026-11-02 22:43:48.454997936 +0000 UTC m=+2264.719991155" Nov 02 22:43:56.490418 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:43:56.490378 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:43:59.230671 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:59.230640 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:43:59.231190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:43:59.230722 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:44:16.054651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:16.054608 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5"] Nov 02 22:44:16.104420 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:16.104379 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pl6h5"] Nov 02 22:44:17.309284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:17.309250 5109 scope.go:122] "RemoveContainer" containerID="5e1c9e482f7f60b4ba4abaf467686fd277b1be8eb900b70408d273f725104177" Nov 02 22:44:17.318513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:17.318491 5109 scope.go:122] "RemoveContainer" containerID="efcefd6d9143732ebb405665c8ed3c9bc7bb371dcdaa71c080fd321be3b96c58" Nov 02 22:44:17.890915 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:17.890882 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba6c883-4208-4ec7-abc0-9f8ce0df0b39" path="/var/lib/kubelet/pods/1ba6c883-4208-4ec7-abc0-9f8ce0df0b39/volumes" Nov 02 22:44:26.491246 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:44:26.491206 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:44:44.047715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:44.047680 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x"] Nov 02 22:44:44.097745 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:44.097707 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2rv2x"] Nov 02 22:44:45.891354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:45.891309 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f63c140b-171d-4193-b2ec-50ca0e30005d" path="/var/lib/kubelet/pods/f63c140b-171d-4193-b2ec-50ca0e30005d/volumes" Nov 02 22:44:46.266552 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.266514 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72"] Nov 02 22:44:46.270205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.270183 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.323827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.323793 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:44:46.326395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.326358 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72"] Nov 02 22:44:46.340200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.340169 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.340200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.340208 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.340412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.340228 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvjd\" (UniqueName: \"kubernetes.io/projected/eff5ac10-c100-408e-a450-199b6cbf5a8c-kube-api-access-2dvjd\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.373202 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.373168 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:44:46.441560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.441519 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.441763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.441561 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.441763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.441613 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvjd\" (UniqueName: \"kubernetes.io/projected/eff5ac10-c100-408e-a450-199b6cbf5a8c-kube-api-access-2dvjd\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.442173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.442152 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.444131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.444098 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.503212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.503179 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvjd\" (UniqueName: \"kubernetes.io/projected/eff5ac10-c100-408e-a450-199b6cbf5a8c-kube-api-access-2dvjd\") pod \"maas-api-revoke-sub-9d7d3240-9jk72\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.583425 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.583339 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:46.953070 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:44:46.953026 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff5ac10_c100_408e_a450_199b6cbf5a8c.slice/crio-023c2a56462f837f46da46847ffd461ce75bfb9191a05e060baf7191a64ae33c WatchSource:0}: Error finding container 023c2a56462f837f46da46847ffd461ce75bfb9191a05e060baf7191a64ae33c: Status 404 returned error can't find the container with id 023c2a56462f837f46da46847ffd461ce75bfb9191a05e060baf7191a64ae33c Nov 02 22:44:46.954686 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:46.954560 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72"] Nov 02 22:44:47.540707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:47.540670 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" event={"ID":"eff5ac10-c100-408e-a450-199b6cbf5a8c","Type":"ContainerStarted","Data":"f3947cb24ac0ed607f397e106b3025b3dca10cf9816ebb97cff5464bd4be7c27"} Nov 02 22:44:47.540911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:47.540726 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" event={"ID":"eff5ac10-c100-408e-a450-199b6cbf5a8c","Type":"ContainerStarted","Data":"023c2a56462f837f46da46847ffd461ce75bfb9191a05e060baf7191a64ae33c"} Nov 02 22:44:47.657623 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:47.657546 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" podStartSLOduration=1.6575304499999999 podStartE2EDuration="1.65753045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:47.65589513 +0000 UTC m=+2323.920888351" watchObservedRunningTime="2026-11-02 22:44:47.65753045 +0000 UTC m=+2323.922523669" Nov 02 22:44:48.547715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:48.547612 5109 generic.go:356] "Generic (PLEG): container finished" podID="eff5ac10-c100-408e-a450-199b6cbf5a8c" containerID="f3947cb24ac0ed607f397e106b3025b3dca10cf9816ebb97cff5464bd4be7c27" exitCode=0 Nov 02 22:44:48.548146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:48.547737 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" event={"ID":"eff5ac10-c100-408e-a450-199b6cbf5a8c","Type":"ContainerDied","Data":"f3947cb24ac0ed607f397e106b3025b3dca10cf9816ebb97cff5464bd4be7c27"} Nov 02 22:44:49.697834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.697806 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:49.774250 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.774213 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-openshift-ca\") pod \"eff5ac10-c100-408e-a450-199b6cbf5a8c\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " Nov 02 22:44:49.774429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.774261 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff5ac10-c100-408e-a450-199b6cbf5a8c-kube-api-access-2dvjd\" (UniqueName: \"kubernetes.io/projected/eff5ac10-c100-408e-a450-199b6cbf5a8c-kube-api-access-2dvjd\") pod \"eff5ac10-c100-408e-a450-199b6cbf5a8c\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " Nov 02 22:44:49.774429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.774390 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-xks-ca\") pod \"eff5ac10-c100-408e-a450-199b6cbf5a8c\" (UID: \"eff5ac10-c100-408e-a450-199b6cbf5a8c\") " Nov 02 22:44:49.774580 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.774541 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-openshift-ca" pod "eff5ac10-c100-408e-a450-199b6cbf5a8c" (UID: "eff5ac10-c100-408e-a450-199b6cbf5a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:44:49.774761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.774741 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:49.776607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.776584 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-xks-ca" pod "eff5ac10-c100-408e-a450-199b6cbf5a8c" (UID: "eff5ac10-c100-408e-a450-199b6cbf5a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:44:49.776722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.776647 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff5ac10-c100-408e-a450-199b6cbf5a8c-kube-api-access-2dvjd" pod "eff5ac10-c100-408e-a450-199b6cbf5a8c" (UID: "eff5ac10-c100-408e-a450-199b6cbf5a8c"). InnerVolumeSpecName "kube-api-access-2dvjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:44:49.875433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.875342 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff5ac10-c100-408e-a450-199b6cbf5a8c-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:49.875433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:49.875372 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvjd\" (UniqueName: \"kubernetes.io/projected/eff5ac10-c100-408e-a450-199b6cbf5a8c-kube-api-access-2dvjd\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:50.558812 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:50.558777 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" Nov 02 22:44:50.559006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:50.558810 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72" event={"ID":"eff5ac10-c100-408e-a450-199b6cbf5a8c","Type":"ContainerDied","Data":"023c2a56462f837f46da46847ffd461ce75bfb9191a05e060baf7191a64ae33c"} Nov 02 22:44:50.559006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:50.558887 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="023c2a56462f837f46da46847ffd461ce75bfb9191a05e060baf7191a64ae33c" Nov 02 22:44:56.490040 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:44:56.489999 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:44:58.565807 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.565770 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp"] Nov 02 22:44:58.566312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.566202 5109 cpu_manager.go:414] "containerMap: removing container" podUID="eff5ac10-c100-408e-a450-199b6cbf5a8c" containerName="revoke-keys" Nov 02 22:44:58.566312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.566213 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff5ac10-c100-408e-a450-199b6cbf5a8c" containerName="revoke-keys" Nov 02 22:44:58.566312 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.566299 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff5ac10-c100-408e-a450-199b6cbf5a8c" containerName="revoke-keys" Nov 02 22:44:58.569602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.569561 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.623429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.623168 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:44:58.663377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.663335 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.663604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.663449 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpf46\" (UniqueName: \"kubernetes.io/projected/41aac931-8a3a-4292-a0e9-ccda552c79a9-kube-api-access-cpf46\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.663604 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.663582 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.673136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.673099 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:44:58.675231 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.675204 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp"] Nov 02 22:44:58.765091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.765051 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpf46\" (UniqueName: \"kubernetes.io/projected/41aac931-8a3a-4292-a0e9-ccda552c79a9-kube-api-access-cpf46\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.765325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.765120 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.765325 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.765203 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.765899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.765874 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.767846 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.767819 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.826702 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.826626 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpf46\" (UniqueName: \"kubernetes.io/projected/41aac931-8a3a-4292-a0e9-ccda552c79a9-kube-api-access-cpf46\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:58.882391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:58.882353 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:44:59.256004 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:44:59.255969 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41aac931_8a3a_4292_a0e9_ccda552c79a9.slice/crio-81262617d34e1bf4e70dfacdb1c5083d27326e0814383e20c5bed9d5aec9a651 WatchSource:0}: Error finding container 81262617d34e1bf4e70dfacdb1c5083d27326e0814383e20c5bed9d5aec9a651: Status 404 returned error can't find the container with id 81262617d34e1bf4e70dfacdb1c5083d27326e0814383e20c5bed9d5aec9a651 Nov 02 22:44:59.257031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:59.256996 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp"] Nov 02 22:44:59.613751 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:59.613715 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" event={"ID":"41aac931-8a3a-4292-a0e9-ccda552c79a9","Type":"ContainerStarted","Data":"33c7096f1f6ebb2de60c4faf7f5e69f2868091bcf4bf61b4f931a1b78f6ea98a"} Nov 02 22:44:59.614160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:59.613769 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" event={"ID":"41aac931-8a3a-4292-a0e9-ccda552c79a9","Type":"ContainerStarted","Data":"81262617d34e1bf4e70dfacdb1c5083d27326e0814383e20c5bed9d5aec9a651"} Nov 02 22:44:59.731519 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:44:59.731367 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" podStartSLOduration=1.731346716 podStartE2EDuration="1.731346716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:59.729749951 +0000 UTC m=+2335.994743171" watchObservedRunningTime="2026-11-02 22:44:59.731346716 +0000 UTC m=+2335.996339935" Nov 02 22:45:00.162877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.162844 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5"] Nov 02 22:45:00.166626 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.166604 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.259834 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.259798 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5"] Nov 02 22:45:00.280226 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.280191 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhjn\" (UniqueName: \"kubernetes.io/projected/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-kube-api-access-pfhjn\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.280428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.280256 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.381739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.381634 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhjn\" (UniqueName: \"kubernetes.io/projected/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-kube-api-access-pfhjn\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.381739 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.381690 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.382346 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.382321 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.443764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.443725 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhjn\" (UniqueName: \"kubernetes.io/projected/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-kube-api-access-pfhjn\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.520508 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.520468 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:00.621244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.621208 5109 generic.go:356] "Generic (PLEG): container finished" podID="41aac931-8a3a-4292-a0e9-ccda552c79a9" containerID="33c7096f1f6ebb2de60c4faf7f5e69f2868091bcf4bf61b4f931a1b78f6ea98a" exitCode=0 Nov 02 22:45:00.621715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.621250 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" event={"ID":"41aac931-8a3a-4292-a0e9-ccda552c79a9","Type":"ContainerDied","Data":"33c7096f1f6ebb2de60c4faf7f5e69f2868091bcf4bf61b4f931a1b78f6ea98a"} Nov 02 22:45:00.915647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:00.914856 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5"] Nov 02 22:45:00.921410 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:45:00.921369 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a687ec_6988_4e6c_a8b7_bd0e5fe45026.slice/crio-a13a38413b5883e8bfa8a5f445d80f45b12274fdb4fc01bac4fbd6616309f279 WatchSource:0}: Error finding container a13a38413b5883e8bfa8a5f445d80f45b12274fdb4fc01bac4fbd6616309f279: Status 404 returned error can't find the container with id a13a38413b5883e8bfa8a5f445d80f45b12274fdb4fc01bac4fbd6616309f279 Nov 02 22:45:01.629465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.629410 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" event={"ID":"90a687ec-6988-4e6c-a8b7-bd0e5fe45026","Type":"ContainerStarted","Data":"8a2cca0538681f863b08dafd4aacb8172f6bae3c882ec7868275db9a54bd8d6a"} Nov 02 22:45:01.630160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.629492 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" event={"ID":"90a687ec-6988-4e6c-a8b7-bd0e5fe45026","Type":"ContainerStarted","Data":"a13a38413b5883e8bfa8a5f445d80f45b12274fdb4fc01bac4fbd6616309f279"} Nov 02 22:45:01.744941 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.744883 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" podStartSLOduration=1.7448696620000002 podStartE2EDuration="1.744869662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:01.743920205 +0000 UTC m=+2338.008913437" watchObservedRunningTime="2026-11-02 22:45:01.744869662 +0000 UTC m=+2338.009862880" Nov 02 22:45:01.784736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.784701 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:45:01.899660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.899556 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-xks-ca\") pod \"41aac931-8a3a-4292-a0e9-ccda552c79a9\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " Nov 02 22:45:01.899819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.899710 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41aac931-8a3a-4292-a0e9-ccda552c79a9-kube-api-access-cpf46\" (UniqueName: \"kubernetes.io/projected/41aac931-8a3a-4292-a0e9-ccda552c79a9-kube-api-access-cpf46\") pod \"41aac931-8a3a-4292-a0e9-ccda552c79a9\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " Nov 02 22:45:01.899819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.899731 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-openshift-ca\") pod \"41aac931-8a3a-4292-a0e9-ccda552c79a9\" (UID: \"41aac931-8a3a-4292-a0e9-ccda552c79a9\") " Nov 02 22:45:01.900141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.900118 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-openshift-ca" pod "41aac931-8a3a-4292-a0e9-ccda552c79a9" (UID: "41aac931-8a3a-4292-a0e9-ccda552c79a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:01.901820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.901800 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-xks-ca" pod "41aac931-8a3a-4292-a0e9-ccda552c79a9" (UID: "41aac931-8a3a-4292-a0e9-ccda552c79a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:45:01.901985 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:01.901963 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41aac931-8a3a-4292-a0e9-ccda552c79a9-kube-api-access-cpf46" pod "41aac931-8a3a-4292-a0e9-ccda552c79a9" (UID: "41aac931-8a3a-4292-a0e9-ccda552c79a9"). InnerVolumeSpecName "kube-api-access-cpf46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:02.001094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.001049 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpf46\" (UniqueName: \"kubernetes.io/projected/41aac931-8a3a-4292-a0e9-ccda552c79a9-kube-api-access-cpf46\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:02.001094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.001085 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:02.001094 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.001096 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41aac931-8a3a-4292-a0e9-ccda552c79a9-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:02.636703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.636668 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" Nov 02 22:45:02.637203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.636667 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp" event={"ID":"41aac931-8a3a-4292-a0e9-ccda552c79a9","Type":"ContainerDied","Data":"81262617d34e1bf4e70dfacdb1c5083d27326e0814383e20c5bed9d5aec9a651"} Nov 02 22:45:02.637203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.636812 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81262617d34e1bf4e70dfacdb1c5083d27326e0814383e20c5bed9d5aec9a651" Nov 02 22:45:02.638390 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.638365 5109 generic.go:356] "Generic (PLEG): container finished" podID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerID="8a2cca0538681f863b08dafd4aacb8172f6bae3c882ec7868275db9a54bd8d6a" exitCode=60 Nov 02 22:45:02.638513 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.638457 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" event={"ID":"90a687ec-6988-4e6c-a8b7-bd0e5fe45026","Type":"ContainerDied","Data":"8a2cca0538681f863b08dafd4aacb8172f6bae3c882ec7868275db9a54bd8d6a"} Nov 02 22:45:02.638720 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:02.638705 5109 scope.go:122] "RemoveContainer" containerID="8a2cca0538681f863b08dafd4aacb8172f6bae3c882ec7868275db9a54bd8d6a" Nov 02 22:45:03.645975 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:03.645878 5109 generic.go:356] "Generic (PLEG): container finished" podID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerID="e112e75ae70b2679831d46d306a7f40c547192140c48af7c76fc64ef020ab38f" exitCode=60 Nov 02 22:45:03.646521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:03.645967 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" event={"ID":"90a687ec-6988-4e6c-a8b7-bd0e5fe45026","Type":"ContainerDied","Data":"e112e75ae70b2679831d46d306a7f40c547192140c48af7c76fc64ef020ab38f"} Nov 02 22:45:03.646521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:03.646038 5109 scope.go:122] "RemoveContainer" containerID="8a2cca0538681f863b08dafd4aacb8172f6bae3c882ec7868275db9a54bd8d6a" Nov 02 22:45:03.646521 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:03.646376 5109 scope.go:122] "RemoveContainer" containerID="e112e75ae70b2679831d46d306a7f40c547192140c48af7c76fc64ef020ab38f" Nov 02 22:45:03.646752 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:45:03.646725 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5_odh-ai-gateway-infra(90a687ec-6988-4e6c-a8b7-bd0e5fe45026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" Nov 02 22:45:04.654512 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:04.654483 5109 scope.go:122] "RemoveContainer" containerID="e112e75ae70b2679831d46d306a7f40c547192140c48af7c76fc64ef020ab38f" Nov 02 22:45:04.654936 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:45:04.654741 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5_odh-ai-gateway-infra(90a687ec-6988-4e6c-a8b7-bd0e5fe45026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" Nov 02 22:45:10.074779 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.074741 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj"] Nov 02 22:45:10.180222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.180182 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm"] Nov 02 22:45:10.499990 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.499899 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5"] Nov 02 22:45:10.666374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.666347 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:10.689173 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.689145 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" Nov 02 22:45:10.689386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.689181 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5" event={"ID":"90a687ec-6988-4e6c-a8b7-bd0e5fe45026","Type":"ContainerDied","Data":"a13a38413b5883e8bfa8a5f445d80f45b12274fdb4fc01bac4fbd6616309f279"} Nov 02 22:45:10.689386 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.689247 5109 scope.go:122] "RemoveContainer" containerID="e112e75ae70b2679831d46d306a7f40c547192140c48af7c76fc64ef020ab38f" Nov 02 22:45:10.819359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.819325 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-kube-api-access-pfhjn\" (UniqueName: \"kubernetes.io/projected/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-kube-api-access-pfhjn\") pod \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " Nov 02 22:45:10.819359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.819367 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-maas-api-service-ca\") pod \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\" (UID: \"90a687ec-6988-4e6c-a8b7-bd0e5fe45026\") " Nov 02 22:45:10.819761 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.819737 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-maas-api-service-ca" pod "90a687ec-6988-4e6c-a8b7-bd0e5fe45026" (UID: "90a687ec-6988-4e6c-a8b7-bd0e5fe45026"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:10.825615 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.825546 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-kube-api-access-pfhjn" pod "90a687ec-6988-4e6c-a8b7-bd0e5fe45026" (UID: "90a687ec-6988-4e6c-a8b7-bd0e5fe45026"). InnerVolumeSpecName "kube-api-access-pfhjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:10.920823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.920784 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfhjn\" (UniqueName: \"kubernetes.io/projected/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-kube-api-access-pfhjn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:10.920823 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:10.920815 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a687ec-6988-4e6c-a8b7-bd0e5fe45026-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:11.110340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:11.110244 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5"] Nov 02 22:45:11.114999 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:11.114968 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a8baae-2989432qbbj5"] Nov 02 22:45:11.891843 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:11.891805 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" path="/var/lib/kubelet/pods/90a687ec-6988-4e6c-a8b7-bd0e5fe45026/volumes" Nov 02 22:45:14.994916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:14.994876 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp"] Nov 02 22:45:15.044808 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:15.044777 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a8baae-kmvjp"] Nov 02 22:45:15.049007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:15.048978 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr"] Nov 02 22:45:15.051821 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:15.051797 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-47wjr"] Nov 02 22:45:15.075677 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:15.075644 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" podUID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" containerName="payload-processing" containerID="cri-o://70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386" gracePeriod=25 Nov 02 22:45:15.181111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:15.181061 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" podUID="e50e1106-768b-4ba5-9961-d22df26b8369" containerName="payload-pre-processing" containerID="cri-o://729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3" gracePeriod=25 Nov 02 22:45:15.899365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:15.899327 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ab469b-36fd-4d32-aab4-caa7f33d08e4" path="/var/lib/kubelet/pods/38ab469b-36fd-4d32-aab4-caa7f33d08e4/volumes" Nov 02 22:45:15.899847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:15.899830 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41aac931-8a3a-4292-a0e9-ccda552c79a9" path="/var/lib/kubelet/pods/41aac931-8a3a-4292-a0e9-ccda552c79a9/volumes" Nov 02 22:45:17.345181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:17.345148 5109 scope.go:122] "RemoveContainer" containerID="342056b255bf67dc25c851e5016a4406ef8367e8d4737c9893b6bfafd7a16658" Nov 02 22:45:17.354481 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:17.354460 5109 scope.go:122] "RemoveContainer" containerID="bedef08c3c346824457a45477ef1cf7a0be596803a49f39ba81fb942ee4d5b47" Nov 02 22:45:17.363317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:17.363298 5109 scope.go:122] "RemoveContainer" containerID="74fa02739f3de024acabbb958509edba7a3ec2b36c5c6cd221b505f105b68df7" Nov 02 22:45:19.228415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:19.228379 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" podUID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:45:19.228875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:19.228379 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" podUID="e50e1106-768b-4ba5-9961-d22df26b8369" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:45:23.827715 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.827681 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k"] Nov 02 22:45:23.878069 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.878035 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2k8k"] Nov 02 22:45:23.890413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.890377 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f3a192-db58-4e00-8e61-2290273186c4" path="/var/lib/kubelet/pods/c1f3a192-db58-4e00-8e61-2290273186c4/volumes" Nov 02 22:45:23.953811 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.953772 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c"] Nov 02 22:45:23.954269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.954250 5109 cpu_manager.go:414] "containerMap: removing container" podUID="41aac931-8a3a-4292-a0e9-ccda552c79a9" containerName="revoke-keys" Nov 02 22:45:23.954365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.954273 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41aac931-8a3a-4292-a0e9-ccda552c79a9" containerName="revoke-keys" Nov 02 22:45:23.954365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.954295 5109 cpu_manager.go:414] "containerMap: removing container" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerName="cleanup" Nov 02 22:45:23.954365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.954303 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerName="cleanup" Nov 02 22:45:23.954518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.954428 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerName="cleanup" Nov 02 22:45:23.954518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.954451 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="41aac931-8a3a-4292-a0e9-ccda552c79a9" containerName="revoke-keys" Nov 02 22:45:23.959286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:23.959264 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.012305 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.012277 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:45:24.012493 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.012335 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:45:24.012628 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.012606 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c"] Nov 02 22:45:24.048669 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.048639 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5g8l\" (UniqueName: \"kubernetes.io/projected/f3743acb-0bb9-4468-84b2-baaef53c97df-kube-api-access-p5g8l\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.048819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.048762 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.048819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.048795 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.150261 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.150169 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5g8l\" (UniqueName: \"kubernetes.io/projected/f3743acb-0bb9-4468-84b2-baaef53c97df-kube-api-access-p5g8l\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.150429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.150272 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.150429 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.150319 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.150950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.150921 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.152925 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.152898 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.212202 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.212172 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5g8l\" (UniqueName: \"kubernetes.io/projected/f3743acb-0bb9-4468-84b2-baaef53c97df-kube-api-access-p5g8l\") pod \"maas-api-revoke-sub-3d4156f1-9998c\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.272747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.272710 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:24.647470 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:45:24.647438 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3743acb_0bb9_4468_84b2_baaef53c97df.slice/crio-c0b5df0204fbee506999ac19869f634ef85b7e02c4cb78672a023ab6af2ff8c7 WatchSource:0}: Error finding container c0b5df0204fbee506999ac19869f634ef85b7e02c4cb78672a023ab6af2ff8c7: Status 404 returned error can't find the container with id c0b5df0204fbee506999ac19869f634ef85b7e02c4cb78672a023ab6af2ff8c7 Nov 02 22:45:24.648951 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.648924 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c"] Nov 02 22:45:24.788471 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:24.788438 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" event={"ID":"f3743acb-0bb9-4468-84b2-baaef53c97df","Type":"ContainerStarted","Data":"c0b5df0204fbee506999ac19869f634ef85b7e02c4cb78672a023ab6af2ff8c7"} Nov 02 22:45:25.795203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:25.795169 5109 generic.go:356] "Generic (PLEG): container finished" podID="f3743acb-0bb9-4468-84b2-baaef53c97df" containerID="9bc076bff2387949c78a517d887e81ba57a2812f4d36cbb9d701705d2505ba33" exitCode=0 Nov 02 22:45:25.795674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:25.795225 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" event={"ID":"f3743acb-0bb9-4468-84b2-baaef53c97df","Type":"ContainerDied","Data":"9bc076bff2387949c78a517d887e81ba57a2812f4d36cbb9d701705d2505ba33"} Nov 02 22:45:26.490352 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:45:26.490309 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:45:26.947372 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:26.947346 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:26.980342 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:26.980306 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-xks-ca\") pod \"f3743acb-0bb9-4468-84b2-baaef53c97df\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " Nov 02 22:45:26.980531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:26.980359 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-openshift-ca\") pod \"f3743acb-0bb9-4468-84b2-baaef53c97df\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " Nov 02 22:45:26.980531 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:26.980439 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3743acb-0bb9-4468-84b2-baaef53c97df-kube-api-access-p5g8l\" (UniqueName: \"kubernetes.io/projected/f3743acb-0bb9-4468-84b2-baaef53c97df-kube-api-access-p5g8l\") pod \"f3743acb-0bb9-4468-84b2-baaef53c97df\" (UID: \"f3743acb-0bb9-4468-84b2-baaef53c97df\") " Nov 02 22:45:26.980864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:26.980838 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-openshift-ca" pod "f3743acb-0bb9-4468-84b2-baaef53c97df" (UID: "f3743acb-0bb9-4468-84b2-baaef53c97df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:26.982771 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:26.982751 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3743acb-0bb9-4468-84b2-baaef53c97df-kube-api-access-p5g8l" pod "f3743acb-0bb9-4468-84b2-baaef53c97df" (UID: "f3743acb-0bb9-4468-84b2-baaef53c97df"). InnerVolumeSpecName "kube-api-access-p5g8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:26.982865 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:26.982846 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-xks-ca" pod "f3743acb-0bb9-4468-84b2-baaef53c97df" (UID: "f3743acb-0bb9-4468-84b2-baaef53c97df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:45:27.082108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:27.082028 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5g8l\" (UniqueName: \"kubernetes.io/projected/f3743acb-0bb9-4468-84b2-baaef53c97df-kube-api-access-p5g8l\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.082108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:27.082063 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.082108 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:27.082073 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3743acb-0bb9-4468-84b2-baaef53c97df-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.806824 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:27.806782 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" event={"ID":"f3743acb-0bb9-4468-84b2-baaef53c97df","Type":"ContainerDied","Data":"c0b5df0204fbee506999ac19869f634ef85b7e02c4cb78672a023ab6af2ff8c7"} Nov 02 22:45:27.807023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:27.806875 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0b5df0204fbee506999ac19869f634ef85b7e02c4cb78672a023ab6af2ff8c7" Nov 02 22:45:27.807023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:27.806798 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c" Nov 02 22:45:29.228806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:29.228767 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" podUID="e50e1106-768b-4ba5-9961-d22df26b8369" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:45:29.228806 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:29.228798 5109 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" podUID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:45:35.360061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.360033 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:45:35.433942 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.433913 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:45:35.470689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.470656 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-kube-api-access-4b8rt\" (UniqueName: \"kubernetes.io/projected/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-kube-api-access-4b8rt\") pod \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " Nov 02 22:45:35.470899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.470718 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-plugins-config-volume\") pod \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\" (UID: \"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f\") " Nov 02 22:45:35.471139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.471115 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-plugins-config-volume" pod "538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" (UID: "538dbdd7-c60c-4dd7-9b2a-8877f0d3537f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:35.473152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.473127 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-kube-api-access-4b8rt" pod "538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" (UID: "538dbdd7-c60c-4dd7-9b2a-8877f0d3537f"). InnerVolumeSpecName "kube-api-access-4b8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:35.572131 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.572089 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50e1106-768b-4ba5-9961-d22df26b8369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50e1106-768b-4ba5-9961-d22df26b8369-plugins-config-volume\") pod \"e50e1106-768b-4ba5-9961-d22df26b8369\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " Nov 02 22:45:35.572334 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.572171 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50e1106-768b-4ba5-9961-d22df26b8369-kube-api-access-mh9zn\" (UniqueName: \"kubernetes.io/projected/e50e1106-768b-4ba5-9961-d22df26b8369-kube-api-access-mh9zn\") pod \"e50e1106-768b-4ba5-9961-d22df26b8369\" (UID: \"e50e1106-768b-4ba5-9961-d22df26b8369\") " Nov 02 22:45:35.572402 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.572389 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:35.572448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.572403 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8rt\" (UniqueName: \"kubernetes.io/projected/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f-kube-api-access-4b8rt\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:35.572656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.572627 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50e1106-768b-4ba5-9961-d22df26b8369-plugins-config-volume" pod "e50e1106-768b-4ba5-9961-d22df26b8369" (UID: "e50e1106-768b-4ba5-9961-d22df26b8369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:35.574496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.574477 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50e1106-768b-4ba5-9961-d22df26b8369-kube-api-access-mh9zn" pod "e50e1106-768b-4ba5-9961-d22df26b8369" (UID: "e50e1106-768b-4ba5-9961-d22df26b8369"). InnerVolumeSpecName "kube-api-access-mh9zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:35.673211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.673095 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh9zn\" (UniqueName: \"kubernetes.io/projected/e50e1106-768b-4ba5-9961-d22df26b8369-kube-api-access-mh9zn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:35.673211 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.673137 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50e1106-768b-4ba5-9961-d22df26b8369-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:35.851778 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.851741 5109 generic.go:356] "Generic (PLEG): container finished" podID="e50e1106-768b-4ba5-9961-d22df26b8369" containerID="729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3" exitCode=0 Nov 02 22:45:35.852000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.851799 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" Nov 02 22:45:35.852000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.851832 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" event={"ID":"e50e1106-768b-4ba5-9961-d22df26b8369","Type":"ContainerDied","Data":"729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3"} Nov 02 22:45:35.852000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.851896 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm" event={"ID":"e50e1106-768b-4ba5-9961-d22df26b8369","Type":"ContainerDied","Data":"0c7d3edf50c8fdb131e136d14d84bb6c56ca300d21545d5da6a68451bc49236d"} Nov 02 22:45:35.852000 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.851929 5109 scope.go:122] "RemoveContainer" containerID="729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3" Nov 02 22:45:35.853382 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.853362 5109 generic.go:356] "Generic (PLEG): container finished" podID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" containerID="70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386" exitCode=0 Nov 02 22:45:35.853499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.853399 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" event={"ID":"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f","Type":"ContainerDied","Data":"70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386"} Nov 02 22:45:35.853499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.853466 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" event={"ID":"538dbdd7-c60c-4dd7-9b2a-8877f0d3537f","Type":"ContainerDied","Data":"1bfa758713eb9e6a0ce823277362a0547a0cb5a3f4d8dc55e2de4b6183a76bb9"} Nov 02 22:45:35.853499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.853417 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj" Nov 02 22:45:35.861451 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.861430 5109 scope.go:122] "RemoveContainer" containerID="729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3" Nov 02 22:45:35.861785 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:45:35.861760 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3\": container with ID starting with 729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3 not found: ID does not exist" containerID="729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3" Nov 02 22:45:35.861883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.861793 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3"} err="failed to get container status \"729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3\": rpc error: code = NotFound desc = could not find container \"729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3\": container with ID starting with 729fbf17bde2a630111fc50d9f0c634fa255f523d7db8bbc69424bb29b275de3 not found: ID does not exist" Nov 02 22:45:35.861883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.861812 5109 scope.go:122] "RemoveContainer" containerID="70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386" Nov 02 22:45:35.870464 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.870446 5109 scope.go:122] "RemoveContainer" containerID="70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386" Nov 02 22:45:35.870794 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:45:35.870776 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386\": container with ID starting with 70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386 not found: ID does not exist" containerID="70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386" Nov 02 22:45:35.870853 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:35.870803 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386"} err="failed to get container status \"70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386\": rpc error: code = NotFound desc = could not find container \"70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386\": container with ID starting with 70ff262ad45f46a4de983d00ae1d2379df1e1d748010853ea20ee39505a39386 not found: ID does not exist" Nov 02 22:45:36.017335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:36.017300 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj"] Nov 02 22:45:36.021383 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:36.021358 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a8baae-cff58fc75tr2mj"] Nov 02 22:45:36.132036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:36.132003 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm"] Nov 02 22:45:36.138475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:36.138447 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a8baae-785d6s4vzm"] Nov 02 22:45:37.890539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:37.890503 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" path="/var/lib/kubelet/pods/538dbdd7-c60c-4dd7-9b2a-8877f0d3537f/volumes" Nov 02 22:45:37.890979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:37.890928 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50e1106-768b-4ba5-9961-d22df26b8369" path="/var/lib/kubelet/pods/e50e1106-768b-4ba5-9961-d22df26b8369/volumes" Nov 02 22:45:39.052593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:39.052259 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5"] Nov 02 22:45:39.105597 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:39.101922 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9tsr5"] Nov 02 22:45:39.901594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:45:39.900884 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a791be60-80ac-444c-b200-1ce76fcf0594" path="/var/lib/kubelet/pods/a791be60-80ac-444c-b200-1ce76fcf0594/volumes" Nov 02 22:45:56.490745 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:45:56.490705 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:46:17.396121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:46:17.396085 5109 scope.go:122] "RemoveContainer" containerID="2f1da0bff88d2cfa1c84bc391b60e01d040dd4665270cac4ec69afb568751f85" Nov 02 22:46:17.404922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:46:17.404898 5109 scope.go:122] "RemoveContainer" containerID="6732a4d19c6a3543905470d10952f2d6c8084d175d2d2169b63de3026e5562f3" Nov 02 22:46:26.490608 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:46:26.490545 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:46:56.490317 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:46:56.490206 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:47:17.438218 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:17.438185 5109 scope.go:122] "RemoveContainer" containerID="e2965f5621af5c8a470862f3a967eda72a2addad888df1271bdff709d28fb241" Nov 02 22:47:17.447095 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:17.447072 5109 scope.go:122] "RemoveContainer" containerID="80cbbe93e0ed46b5929eac5213133063f3de8e4362234c9725e1172ba703ef87" Nov 02 22:47:22.050377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:22.050339 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv"] Nov 02 22:47:22.100178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:22.100136 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g4wxv"] Nov 02 22:47:23.897543 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:23.897508 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7301b52b-2168-45b6-b6a6-6a5ff794b1af" path="/var/lib/kubelet/pods/7301b52b-2168-45b6-b6a6-6a5ff794b1af/volumes" Nov 02 22:47:26.490821 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:47:26.490775 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:47:31.920345 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.920306 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh"] Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.921989 5109 cpu_manager.go:414] "containerMap: removing container" podUID="f3743acb-0bb9-4468-84b2-baaef53c97df" containerName="revoke-keys" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922033 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3743acb-0bb9-4468-84b2-baaef53c97df" containerName="revoke-keys" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922105 5109 cpu_manager.go:414] "containerMap: removing container" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerName="cleanup" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922114 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerName="cleanup" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922147 5109 cpu_manager.go:414] "containerMap: removing container" podUID="e50e1106-768b-4ba5-9961-d22df26b8369" containerName="payload-pre-processing" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922155 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50e1106-768b-4ba5-9961-d22df26b8369" containerName="payload-pre-processing" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922188 5109 cpu_manager.go:414] "containerMap: removing container" podUID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" containerName="payload-processing" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922195 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" containerName="payload-processing" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922387 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="538dbdd7-c60c-4dd7-9b2a-8877f0d3537f" containerName="payload-processing" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922402 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3743acb-0bb9-4468-84b2-baaef53c97df" containerName="revoke-keys" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922427 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a687ec-6988-4e6c-a8b7-bd0e5fe45026" containerName="cleanup" Nov 02 22:47:31.924805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.922452 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50e1106-768b-4ba5-9961-d22df26b8369" containerName="payload-pre-processing" Nov 02 22:47:31.928950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.928920 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:31.979780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:31.979734 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh"] Nov 02 22:47:32.006100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.006055 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.006100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.006105 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqspb\" (UniqueName: \"kubernetes.io/projected/9884aa2e-c612-45aa-8004-2e1f2600d21e-kube-api-access-tqspb\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.006379 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.006225 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.027884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.027848 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:47:32.027884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.027871 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:47:32.051510 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.051473 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv"] Nov 02 22:47:32.056956 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.056928 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-q42qv"] Nov 02 22:47:32.107035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.107000 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.107256 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.107048 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqspb\" (UniqueName: \"kubernetes.io/projected/9884aa2e-c612-45aa-8004-2e1f2600d21e-kube-api-access-tqspb\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.107256 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.107086 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.108014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.107985 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.109830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.109811 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.169075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.169043 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqspb\" (UniqueName: \"kubernetes.io/projected/9884aa2e-c612-45aa-8004-2e1f2600d21e-kube-api-access-tqspb\") pod \"maas-api-revoke-sub-d576e68b-kpnrh\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.242385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.242288 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:32.620338 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:47:32.620246 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9884aa2e_c612_45aa_8004_2e1f2600d21e.slice/crio-154528d66d81ca2d56fbd2982cc275324c90cece8a4dda4e8e74c5b788075b2c WatchSource:0}: Error finding container 154528d66d81ca2d56fbd2982cc275324c90cece8a4dda4e8e74c5b788075b2c: Status 404 returned error can't find the container with id 154528d66d81ca2d56fbd2982cc275324c90cece8a4dda4e8e74c5b788075b2c Nov 02 22:47:32.623320 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:32.623286 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh"] Nov 02 22:47:33.517111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:33.517074 5109 generic.go:356] "Generic (PLEG): container finished" podID="9884aa2e-c612-45aa-8004-2e1f2600d21e" containerID="d5c8b600237adb84142c3088290f0fa89bc131a08f4c77c7aafc0e3ef854ee38" exitCode=0 Nov 02 22:47:33.517515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:33.517134 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" event={"ID":"9884aa2e-c612-45aa-8004-2e1f2600d21e","Type":"ContainerDied","Data":"d5c8b600237adb84142c3088290f0fa89bc131a08f4c77c7aafc0e3ef854ee38"} Nov 02 22:47:33.517515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:33.517182 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" event={"ID":"9884aa2e-c612-45aa-8004-2e1f2600d21e","Type":"ContainerStarted","Data":"154528d66d81ca2d56fbd2982cc275324c90cece8a4dda4e8e74c5b788075b2c"} Nov 02 22:47:33.891582 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:33.891539 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26b829b-66af-49d1-a9f4-4d20285ddb08" path="/var/lib/kubelet/pods/b26b829b-66af-49d1-a9f4-4d20285ddb08/volumes" Nov 02 22:47:34.659343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.659318 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:34.834504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.834393 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-openshift-ca\") pod \"9884aa2e-c612-45aa-8004-2e1f2600d21e\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " Nov 02 22:47:34.834723 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.834624 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9884aa2e-c612-45aa-8004-2e1f2600d21e-kube-api-access-tqspb\" (UniqueName: \"kubernetes.io/projected/9884aa2e-c612-45aa-8004-2e1f2600d21e-kube-api-access-tqspb\") pod \"9884aa2e-c612-45aa-8004-2e1f2600d21e\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " Nov 02 22:47:34.834777 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.834726 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-xks-ca\") pod \"9884aa2e-c612-45aa-8004-2e1f2600d21e\" (UID: \"9884aa2e-c612-45aa-8004-2e1f2600d21e\") " Nov 02 22:47:34.834908 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.834885 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-openshift-ca" pod "9884aa2e-c612-45aa-8004-2e1f2600d21e" (UID: "9884aa2e-c612-45aa-8004-2e1f2600d21e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:47:34.835055 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.835042 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:34.836988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.836957 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-xks-ca" pod "9884aa2e-c612-45aa-8004-2e1f2600d21e" (UID: "9884aa2e-c612-45aa-8004-2e1f2600d21e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:47:34.837111 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.837082 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9884aa2e-c612-45aa-8004-2e1f2600d21e-kube-api-access-tqspb" pod "9884aa2e-c612-45aa-8004-2e1f2600d21e" (UID: "9884aa2e-c612-45aa-8004-2e1f2600d21e"). InnerVolumeSpecName "kube-api-access-tqspb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:47:34.936707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.936668 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqspb\" (UniqueName: \"kubernetes.io/projected/9884aa2e-c612-45aa-8004-2e1f2600d21e-kube-api-access-tqspb\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:34.936707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:34.936703 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9884aa2e-c612-45aa-8004-2e1f2600d21e-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:35.529121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:35.529086 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" event={"ID":"9884aa2e-c612-45aa-8004-2e1f2600d21e","Type":"ContainerDied","Data":"154528d66d81ca2d56fbd2982cc275324c90cece8a4dda4e8e74c5b788075b2c"} Nov 02 22:47:35.529121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:35.529119 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh" Nov 02 22:47:35.529121 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:47:35.529137 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154528d66d81ca2d56fbd2982cc275324c90cece8a4dda4e8e74c5b788075b2c" Nov 02 22:47:56.490238 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:47:56.490192 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:48:01.168090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.168054 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt"] Nov 02 22:48:01.168523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.168505 5109 cpu_manager.go:414] "containerMap: removing container" podUID="9884aa2e-c612-45aa-8004-2e1f2600d21e" containerName="revoke-keys" Nov 02 22:48:01.168523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.168517 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9884aa2e-c612-45aa-8004-2e1f2600d21e" containerName="revoke-keys" Nov 02 22:48:01.168635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.168624 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="9884aa2e-c612-45aa-8004-2e1f2600d21e" containerName="revoke-keys" Nov 02 22:48:01.171810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.171789 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.204982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.204947 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8xr\" (UniqueName: \"kubernetes.io/projected/96353aec-7d34-4e37-822c-2e5e058af60d-kube-api-access-ll8xr\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.205181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.205071 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.205181 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.205136 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.276612 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.276544 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:48:01.276799 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.276621 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:48:01.278428 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.278405 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt"] Nov 02 22:48:01.306004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.305962 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8xr\" (UniqueName: \"kubernetes.io/projected/96353aec-7d34-4e37-822c-2e5e058af60d-kube-api-access-ll8xr\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.306203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.306046 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.306203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.306095 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.306824 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.306797 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.308793 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.308770 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.367354 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.367317 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8xr\" (UniqueName: \"kubernetes.io/projected/96353aec-7d34-4e37-822c-2e5e058af60d-kube-api-access-ll8xr\") pod \"maas-api-revoke-sub-d2bc93bc-j25nt\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.485360 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.485271 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:01.855157 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:48:01.855112 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96353aec_7d34_4e37_822c_2e5e058af60d.slice/crio-1db29794790398919e9b56dd09a1f6938ca4fba43da8c7b816ab5cbaa9982e0e WatchSource:0}: Error finding container 1db29794790398919e9b56dd09a1f6938ca4fba43da8c7b816ab5cbaa9982e0e: Status 404 returned error can't find the container with id 1db29794790398919e9b56dd09a1f6938ca4fba43da8c7b816ab5cbaa9982e0e Nov 02 22:48:01.855544 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:01.855457 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt"] Nov 02 22:48:02.666480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:02.666440 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" event={"ID":"96353aec-7d34-4e37-822c-2e5e058af60d","Type":"ContainerStarted","Data":"95b1ca53881159d8a622fc98637ec053107b4de2afcc6636654a9f0aa9e57e3e"} Nov 02 22:48:02.666928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:02.666501 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" event={"ID":"96353aec-7d34-4e37-822c-2e5e058af60d","Type":"ContainerStarted","Data":"1db29794790398919e9b56dd09a1f6938ca4fba43da8c7b816ab5cbaa9982e0e"} Nov 02 22:48:02.783509 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:02.783454 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" podStartSLOduration=1.7834371409999998 podStartE2EDuration="1.783437141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:02.782322206 +0000 UTC m=+2519.047315424" watchObservedRunningTime="2026-11-02 22:48:02.783437141 +0000 UTC m=+2519.048430362" Nov 02 22:48:03.672869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:03.672836 5109 generic.go:356] "Generic (PLEG): container finished" podID="96353aec-7d34-4e37-822c-2e5e058af60d" containerID="95b1ca53881159d8a622fc98637ec053107b4de2afcc6636654a9f0aa9e57e3e" exitCode=0 Nov 02 22:48:03.673284 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:03.672919 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" event={"ID":"96353aec-7d34-4e37-822c-2e5e058af60d","Type":"ContainerDied","Data":"95b1ca53881159d8a622fc98637ec053107b4de2afcc6636654a9f0aa9e57e3e"} Nov 02 22:48:04.815182 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:04.815154 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:04.942090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:04.942016 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96353aec-7d34-4e37-822c-2e5e058af60d-kube-api-access-ll8xr\" (UniqueName: \"kubernetes.io/projected/96353aec-7d34-4e37-822c-2e5e058af60d-kube-api-access-ll8xr\") pod \"96353aec-7d34-4e37-822c-2e5e058af60d\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " Nov 02 22:48:04.942090 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:04.942048 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-xks-ca\") pod \"96353aec-7d34-4e37-822c-2e5e058af60d\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " Nov 02 22:48:04.942308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:04.942098 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-openshift-ca\") pod \"96353aec-7d34-4e37-822c-2e5e058af60d\" (UID: \"96353aec-7d34-4e37-822c-2e5e058af60d\") " Nov 02 22:48:04.942547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:04.942490 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-openshift-ca" pod "96353aec-7d34-4e37-822c-2e5e058af60d" (UID: "96353aec-7d34-4e37-822c-2e5e058af60d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:04.944463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:04.944439 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-xks-ca" pod "96353aec-7d34-4e37-822c-2e5e058af60d" (UID: "96353aec-7d34-4e37-822c-2e5e058af60d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:48:04.944599 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:04.944469 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96353aec-7d34-4e37-822c-2e5e058af60d-kube-api-access-ll8xr" pod "96353aec-7d34-4e37-822c-2e5e058af60d" (UID: "96353aec-7d34-4e37-822c-2e5e058af60d"). InnerVolumeSpecName "kube-api-access-ll8xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:05.042972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:05.042937 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll8xr\" (UniqueName: \"kubernetes.io/projected/96353aec-7d34-4e37-822c-2e5e058af60d-kube-api-access-ll8xr\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:05.042972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:05.042969 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:05.043187 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:05.042983 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96353aec-7d34-4e37-822c-2e5e058af60d-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:05.683932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:05.683894 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" Nov 02 22:48:05.683932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:05.683922 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt" event={"ID":"96353aec-7d34-4e37-822c-2e5e058af60d","Type":"ContainerDied","Data":"1db29794790398919e9b56dd09a1f6938ca4fba43da8c7b816ab5cbaa9982e0e"} Nov 02 22:48:05.684155 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:05.684008 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db29794790398919e9b56dd09a1f6938ca4fba43da8c7b816ab5cbaa9982e0e" Nov 02 22:48:17.471216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:17.471128 5109 scope.go:122] "RemoveContainer" containerID="be0945ac34a901df9d8390a446299a576c8e415cd0340b6dc81319be37830785" Nov 02 22:48:26.490194 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:48:26.490156 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:48:31.162300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.162262 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj"] Nov 02 22:48:31.162790 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.162774 5109 cpu_manager.go:414] "containerMap: removing container" podUID="96353aec-7d34-4e37-822c-2e5e058af60d" containerName="revoke-keys" Nov 02 22:48:31.162837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.162793 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96353aec-7d34-4e37-822c-2e5e058af60d" containerName="revoke-keys" Nov 02 22:48:31.162884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.162875 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="96353aec-7d34-4e37-822c-2e5e058af60d" containerName="revoke-keys" Nov 02 22:48:31.166169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.166149 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.201199 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.201158 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1b6d85-7513-4396-9e0e-de22bc36f51c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.201387 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.201240 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5m7\" (UniqueName: \"kubernetes.io/projected/fb1b6d85-7513-4396-9e0e-de22bc36f51c-kube-api-access-8r5m7\") pod \"payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.272538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.272503 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c86709\"" Nov 02 22:48:31.272928 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.272900 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-768a482a-dockercfg-7nnl9\"" Nov 02 22:48:31.281300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.281246 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj"] Nov 02 22:48:31.306600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.302368 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1b6d85-7513-4396-9e0e-de22bc36f51c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.306600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.302466 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5m7\" (UniqueName: \"kubernetes.io/projected/fb1b6d85-7513-4396-9e0e-de22bc36f51c-kube-api-access-8r5m7\") pod \"payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.306600 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.303760 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1b6d85-7513-4396-9e0e-de22bc36f51c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.331368 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.331334 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6"] Nov 02 22:48:31.336419 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.336388 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.364281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.364234 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5m7\" (UniqueName: \"kubernetes.io/projected/fb1b6d85-7513-4396-9e0e-de22bc36f51c-kube-api-access-8r5m7\") pod \"payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.403330 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.403292 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b742a8f-419e-4bd5-ad74-4e3a7418509a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.403551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.403371 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rzj\" (UniqueName: \"kubernetes.io/projected/1b742a8f-419e-4bd5-ad74-4e3a7418509a-kube-api-access-x5rzj\") pod \"payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.409217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.409185 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6"] Nov 02 22:48:31.479763 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.479670 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:31.504343 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.504315 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rzj\" (UniqueName: \"kubernetes.io/projected/1b742a8f-419e-4bd5-ad74-4e3a7418509a-kube-api-access-x5rzj\") pod \"payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.504514 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.504432 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b742a8f-419e-4bd5-ad74-4e3a7418509a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.505204 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.505183 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b742a8f-419e-4bd5-ad74-4e3a7418509a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.567606 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.565976 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rzj\" (UniqueName: \"kubernetes.io/projected/1b742a8f-419e-4bd5-ad74-4e3a7418509a-kube-api-access-x5rzj\") pod \"payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.654883 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.654849 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:31.854067 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:48:31.854026 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1b6d85_7513_4396_9e0e_de22bc36f51c.slice/crio-abe32586391f1d35f256e9fc0eb960f2b71f33fa991e68356b95b2ee0dca1121 WatchSource:0}: Error finding container abe32586391f1d35f256e9fc0eb960f2b71f33fa991e68356b95b2ee0dca1121: Status 404 returned error can't find the container with id abe32586391f1d35f256e9fc0eb960f2b71f33fa991e68356b95b2ee0dca1121 Nov 02 22:48:31.854271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:31.854247 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj"] Nov 02 22:48:32.025113 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:48:32.025082 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b742a8f_419e_4bd5_ad74_4e3a7418509a.slice/crio-3925165db698ab35c9144c93aa899e90f4d970aa4386084365960d9230821774 WatchSource:0}: Error finding container 3925165db698ab35c9144c93aa899e90f4d970aa4386084365960d9230821774: Status 404 returned error can't find the container with id 3925165db698ab35c9144c93aa899e90f4d970aa4386084365960d9230821774 Nov 02 22:48:32.027418 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.027392 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6"] Nov 02 22:48:32.830611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.830555 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" event={"ID":"fb1b6d85-7513-4396-9e0e-de22bc36f51c","Type":"ContainerStarted","Data":"251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d"} Nov 02 22:48:32.831060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.830759 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:32.831060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.830821 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" event={"ID":"fb1b6d85-7513-4396-9e0e-de22bc36f51c","Type":"ContainerStarted","Data":"abe32586391f1d35f256e9fc0eb960f2b71f33fa991e68356b95b2ee0dca1121"} Nov 02 22:48:32.832479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.832452 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" event={"ID":"1b742a8f-419e-4bd5-ad74-4e3a7418509a","Type":"ContainerStarted","Data":"f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b"} Nov 02 22:48:32.832629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.832501 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:32.832629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.832519 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" event={"ID":"1b742a8f-419e-4bd5-ad74-4e3a7418509a","Type":"ContainerStarted","Data":"3925165db698ab35c9144c93aa899e90f4d970aa4386084365960d9230821774"} Nov 02 22:48:32.949594 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:32.949505 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" podStartSLOduration=1.949485633 podStartE2EDuration="1.949485633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:32.947696794 +0000 UTC m=+2549.212690013" watchObservedRunningTime="2026-11-02 22:48:32.949485633 +0000 UTC m=+2549.214478852" Nov 02 22:48:43.842067 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:43.842024 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:48:43.842527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:43.842401 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:48:43.964880 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:48:43.964822 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" podStartSLOduration=12.964805639 podStartE2EDuration="12.964805639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:33.071389659 +0000 UTC m=+2549.336382879" watchObservedRunningTime="2026-11-02 22:48:43.964805639 +0000 UTC m=+2560.229798859" Nov 02 22:48:56.490852 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:48:56.490814 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:49:17.495947 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:17.495908 5109 scope.go:122] "RemoveContainer" containerID="779d87ff6b4abef256dc099a2ea5d81a6f89b54f1f466ba7c2ba4ed9e61d5a2c" Nov 02 22:49:17.505352 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:17.505329 5109 scope.go:122] "RemoveContainer" containerID="2af07a22d58b91361575b3924ec6d51903e35307c32e2f8a535ec5ea33623454" Nov 02 22:49:17.514241 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:17.514221 5109 scope.go:122] "RemoveContainer" containerID="f52056478f03a503cabbd58b035ceb650e3e4173002a382ce7adf417fe013a2a" Nov 02 22:49:26.489899 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:49:26.489860 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:49:32.495396 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.495355 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k"] Nov 02 22:49:32.499127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.499108 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.602335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.602301 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:49:32.602335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.602336 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:49:32.604541 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.604518 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k"] Nov 02 22:49:32.622363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.622329 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.622547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.622395 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltldg\" (UniqueName: \"kubernetes.io/projected/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-kube-api-access-ltldg\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.622547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.622499 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.723827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.723790 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltldg\" (UniqueName: \"kubernetes.io/projected/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-kube-api-access-ltldg\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.724045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.723840 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.724045 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.723901 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.724624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.724601 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.726355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.726337 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.785740 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.785708 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltldg\" (UniqueName: \"kubernetes.io/projected/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-kube-api-access-ltldg\") pod \"maas-api-revoke-sub-337c8ec2-69q4k\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:32.812873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:32.812836 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:33.184823 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:49:33.184768 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acd53a4_9335_41c6_9c64_03a2e4d1c1da.slice/crio-546e3bc0da8fb6983d440ca06164017cbc4bad28019b25868218c4c29758da60 WatchSource:0}: Error finding container 546e3bc0da8fb6983d440ca06164017cbc4bad28019b25868218c4c29758da60: Status 404 returned error can't find the container with id 546e3bc0da8fb6983d440ca06164017cbc4bad28019b25868218c4c29758da60 Nov 02 22:49:33.186535 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:33.186516 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k"] Nov 02 22:49:34.158042 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:34.158004 5109 generic.go:356] "Generic (PLEG): container finished" podID="2acd53a4-9335-41c6-9c64-03a2e4d1c1da" containerID="a5bce91f69f45c3d47be469f3f53e53f2cbc589442f8f6d89e3300f0f1110b2c" exitCode=0 Nov 02 22:49:34.158492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:34.158091 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" event={"ID":"2acd53a4-9335-41c6-9c64-03a2e4d1c1da","Type":"ContainerDied","Data":"a5bce91f69f45c3d47be469f3f53e53f2cbc589442f8f6d89e3300f0f1110b2c"} Nov 02 22:49:34.158492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:34.158155 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" event={"ID":"2acd53a4-9335-41c6-9c64-03a2e4d1c1da","Type":"ContainerStarted","Data":"546e3bc0da8fb6983d440ca06164017cbc4bad28019b25868218c4c29758da60"} Nov 02 22:49:35.303064 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.303038 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:35.449561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.449481 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-xks-ca\") pod \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " Nov 02 22:49:35.449561 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.449528 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-kube-api-access-ltldg\" (UniqueName: \"kubernetes.io/projected/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-kube-api-access-ltldg\") pod \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " Nov 02 22:49:35.449767 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.449619 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-openshift-ca\") pod \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\" (UID: \"2acd53a4-9335-41c6-9c64-03a2e4d1c1da\") " Nov 02 22:49:35.450021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.449993 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-openshift-ca" pod "2acd53a4-9335-41c6-9c64-03a2e4d1c1da" (UID: "2acd53a4-9335-41c6-9c64-03a2e4d1c1da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:35.451851 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.451830 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-kube-api-access-ltldg" pod "2acd53a4-9335-41c6-9c64-03a2e4d1c1da" (UID: "2acd53a4-9335-41c6-9c64-03a2e4d1c1da"). InnerVolumeSpecName "kube-api-access-ltldg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:35.451940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.451911 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-xks-ca" pod "2acd53a4-9335-41c6-9c64-03a2e4d1c1da" (UID: "2acd53a4-9335-41c6-9c64-03a2e4d1c1da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:35.551318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.551275 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:35.551318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.551312 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:35.551318 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:35.551322 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltldg\" (UniqueName: \"kubernetes.io/projected/2acd53a4-9335-41c6-9c64-03a2e4d1c1da-kube-api-access-ltldg\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:36.170057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:36.169964 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" Nov 02 22:49:36.170057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:36.169977 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k" event={"ID":"2acd53a4-9335-41c6-9c64-03a2e4d1c1da","Type":"ContainerDied","Data":"546e3bc0da8fb6983d440ca06164017cbc4bad28019b25868218c4c29758da60"} Nov 02 22:49:36.170057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:36.170026 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="546e3bc0da8fb6983d440ca06164017cbc4bad28019b25868218c4c29758da60" Nov 02 22:49:44.187824 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.187743 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct"] Nov 02 22:49:44.188238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.188195 5109 cpu_manager.go:414] "containerMap: removing container" podUID="2acd53a4-9335-41c6-9c64-03a2e4d1c1da" containerName="revoke-keys" Nov 02 22:49:44.188238 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.188209 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acd53a4-9335-41c6-9c64-03a2e4d1c1da" containerName="revoke-keys" Nov 02 22:49:44.188340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.188304 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acd53a4-9335-41c6-9c64-03a2e4d1c1da" containerName="revoke-keys" Nov 02 22:49:44.191466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.191445 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.250068 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.250030 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct"] Nov 02 22:49:44.250439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.250267 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:49:44.297784 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.297749 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:49:44.339627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.339542 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lxb\" (UniqueName: \"kubernetes.io/projected/251ed681-3a13-455f-8d15-5c69c2916f21-kube-api-access-t5lxb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.339837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.339695 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.339837 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.339826 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.441141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.441039 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lxb\" (UniqueName: \"kubernetes.io/projected/251ed681-3a13-455f-8d15-5c69c2916f21-kube-api-access-t5lxb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.441141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.441097 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.441359 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.441162 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.441817 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.441798 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.443708 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.443686 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.502511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.502475 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lxb\" (UniqueName: \"kubernetes.io/projected/251ed681-3a13-455f-8d15-5c69c2916f21-kube-api-access-t5lxb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.507323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.507300 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:44.877769 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:49:44.877731 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251ed681_3a13_455f_8d15_5c69c2916f21.slice/crio-30db3f9a9343bf64437ca7a6752e88ef50b2e3b4df9516860625185892924e48 WatchSource:0}: Error finding container 30db3f9a9343bf64437ca7a6752e88ef50b2e3b4df9516860625185892924e48: Status 404 returned error can't find the container with id 30db3f9a9343bf64437ca7a6752e88ef50b2e3b4df9516860625185892924e48 Nov 02 22:49:44.879139 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:44.879113 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct"] Nov 02 22:49:45.222014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:45.221973 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" event={"ID":"251ed681-3a13-455f-8d15-5c69c2916f21","Type":"ContainerStarted","Data":"30db3f9a9343bf64437ca7a6752e88ef50b2e3b4df9516860625185892924e48"} Nov 02 22:49:46.231021 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:46.230913 5109 generic.go:356] "Generic (PLEG): container finished" podID="251ed681-3a13-455f-8d15-5c69c2916f21" containerID="653c1a9937869e84090f1cd3176ccc963a8f280c0673910977e473d2b853140f" exitCode=0 Nov 02 22:49:46.231520 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:46.231014 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" event={"ID":"251ed681-3a13-455f-8d15-5c69c2916f21","Type":"ContainerDied","Data":"653c1a9937869e84090f1cd3176ccc963a8f280c0673910977e473d2b853140f"} Nov 02 22:49:47.380047 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.380015 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:47.473743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.473708 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-openshift-ca\") pod \"251ed681-3a13-455f-8d15-5c69c2916f21\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " Nov 02 22:49:47.473976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.473900 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251ed681-3a13-455f-8d15-5c69c2916f21-kube-api-access-t5lxb\" (UniqueName: \"kubernetes.io/projected/251ed681-3a13-455f-8d15-5c69c2916f21-kube-api-access-t5lxb\") pod \"251ed681-3a13-455f-8d15-5c69c2916f21\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " Nov 02 22:49:47.473976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.473968 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-xks-ca\") pod \"251ed681-3a13-455f-8d15-5c69c2916f21\" (UID: \"251ed681-3a13-455f-8d15-5c69c2916f21\") " Nov 02 22:49:47.474145 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.474114 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-openshift-ca" pod "251ed681-3a13-455f-8d15-5c69c2916f21" (UID: "251ed681-3a13-455f-8d15-5c69c2916f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:47.474357 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.474331 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:47.476165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.476142 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-xks-ca" pod "251ed681-3a13-455f-8d15-5c69c2916f21" (UID: "251ed681-3a13-455f-8d15-5c69c2916f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:47.476276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.476220 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251ed681-3a13-455f-8d15-5c69c2916f21-kube-api-access-t5lxb" pod "251ed681-3a13-455f-8d15-5c69c2916f21" (UID: "251ed681-3a13-455f-8d15-5c69c2916f21"). InnerVolumeSpecName "kube-api-access-t5lxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:47.575800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.575758 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5lxb\" (UniqueName: \"kubernetes.io/projected/251ed681-3a13-455f-8d15-5c69c2916f21-kube-api-access-t5lxb\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:47.575800 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:47.575794 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251ed681-3a13-455f-8d15-5c69c2916f21-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:48.244132 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:48.244051 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" Nov 02 22:49:48.244132 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:48.244094 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct" event={"ID":"251ed681-3a13-455f-8d15-5c69c2916f21","Type":"ContainerDied","Data":"30db3f9a9343bf64437ca7a6752e88ef50b2e3b4df9516860625185892924e48"} Nov 02 22:49:48.244339 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:48.244150 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30db3f9a9343bf64437ca7a6752e88ef50b2e3b4df9516860625185892924e48" Nov 02 22:49:50.048936 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:50.048893 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72"] Nov 02 22:49:50.100370 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:50.100329 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d3240-9jk72"] Nov 02 22:49:51.891376 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:51.891339 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff5ac10-c100-408e-a450-199b6cbf5a8c" path="/var/lib/kubelet/pods/eff5ac10-c100-408e-a450-199b6cbf5a8c/volumes" Nov 02 22:49:55.612588 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:55.612533 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6"] Nov 02 22:49:55.717727 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:49:55.717690 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj"] Nov 02 22:49:56.491420 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:49:56.491379 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:50:00.259273 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.259228 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct"] Nov 02 22:50:00.308897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.308862 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c86709-4z4ct"] Nov 02 22:50:00.613285 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.613154 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" podUID="1b742a8f-419e-4bd5-ad74-4e3a7418509a" containerName="payload-processing" containerID="cri-o://f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b" gracePeriod=25 Nov 02 22:50:00.718950 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.718895 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" podUID="fb1b6d85-7513-4396-9e0e-de22bc36f51c" containerName="payload-pre-processing" containerID="cri-o://251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d" gracePeriod=25 Nov 02 22:50:00.877688 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.877616 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:50:00.902968 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.902929 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b742a8f-419e-4bd5-ad74-4e3a7418509a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b742a8f-419e-4bd5-ad74-4e3a7418509a-plugins-config-volume\") pod \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " Nov 02 22:50:00.903132 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.903075 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b742a8f-419e-4bd5-ad74-4e3a7418509a-kube-api-access-x5rzj\" (UniqueName: \"kubernetes.io/projected/1b742a8f-419e-4bd5-ad74-4e3a7418509a-kube-api-access-x5rzj\") pod \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\" (UID: \"1b742a8f-419e-4bd5-ad74-4e3a7418509a\") " Nov 02 22:50:00.903327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.903305 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b742a8f-419e-4bd5-ad74-4e3a7418509a-plugins-config-volume" pod "1b742a8f-419e-4bd5-ad74-4e3a7418509a" (UID: "1b742a8f-419e-4bd5-ad74-4e3a7418509a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:00.903448 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.903432 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b742a8f-419e-4bd5-ad74-4e3a7418509a-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:00.906100 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.906071 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b742a8f-419e-4bd5-ad74-4e3a7418509a-kube-api-access-x5rzj" pod "1b742a8f-419e-4bd5-ad74-4e3a7418509a" (UID: "1b742a8f-419e-4bd5-ad74-4e3a7418509a"). InnerVolumeSpecName "kube-api-access-x5rzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:00.971412 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:00.971384 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:50:01.004116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.004086 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1b6d85-7513-4396-9e0e-de22bc36f51c-kube-api-access-8r5m7\" (UniqueName: \"kubernetes.io/projected/fb1b6d85-7513-4396-9e0e-de22bc36f51c-kube-api-access-8r5m7\") pod \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " Nov 02 22:50:01.004306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.004202 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1b6d85-7513-4396-9e0e-de22bc36f51c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1b6d85-7513-4396-9e0e-de22bc36f51c-plugins-config-volume\") pod \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\" (UID: \"fb1b6d85-7513-4396-9e0e-de22bc36f51c\") " Nov 02 22:50:01.004552 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.004529 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5rzj\" (UniqueName: \"kubernetes.io/projected/1b742a8f-419e-4bd5-ad74-4e3a7418509a-kube-api-access-x5rzj\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:01.004647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.004594 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1b6d85-7513-4396-9e0e-de22bc36f51c-plugins-config-volume" pod "fb1b6d85-7513-4396-9e0e-de22bc36f51c" (UID: "fb1b6d85-7513-4396-9e0e-de22bc36f51c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:01.006433 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.006416 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1b6d85-7513-4396-9e0e-de22bc36f51c-kube-api-access-8r5m7" pod "fb1b6d85-7513-4396-9e0e-de22bc36f51c" (UID: "fb1b6d85-7513-4396-9e0e-de22bc36f51c"). InnerVolumeSpecName "kube-api-access-8r5m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:01.105190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.105148 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5m7\" (UniqueName: \"kubernetes.io/projected/fb1b6d85-7513-4396-9e0e-de22bc36f51c-kube-api-access-8r5m7\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:01.105190 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.105182 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1b6d85-7513-4396-9e0e-de22bc36f51c-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:01.312649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.312615 5109 generic.go:356] "Generic (PLEG): container finished" podID="fb1b6d85-7513-4396-9e0e-de22bc36f51c" containerID="251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d" exitCode=0 Nov 02 22:50:01.313172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.312663 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" Nov 02 22:50:01.313172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.312703 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" event={"ID":"fb1b6d85-7513-4396-9e0e-de22bc36f51c","Type":"ContainerDied","Data":"251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d"} Nov 02 22:50:01.313172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.312765 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj" event={"ID":"fb1b6d85-7513-4396-9e0e-de22bc36f51c","Type":"ContainerDied","Data":"abe32586391f1d35f256e9fc0eb960f2b71f33fa991e68356b95b2ee0dca1121"} Nov 02 22:50:01.313172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.312795 5109 scope.go:122] "RemoveContainer" containerID="251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d" Nov 02 22:50:01.314219 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.314191 5109 generic.go:356] "Generic (PLEG): container finished" podID="1b742a8f-419e-4bd5-ad74-4e3a7418509a" containerID="f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b" exitCode=0 Nov 02 22:50:01.314344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.314236 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" Nov 02 22:50:01.314344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.314297 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" event={"ID":"1b742a8f-419e-4bd5-ad74-4e3a7418509a","Type":"ContainerDied","Data":"f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b"} Nov 02 22:50:01.314344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.314334 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6" event={"ID":"1b742a8f-419e-4bd5-ad74-4e3a7418509a","Type":"ContainerDied","Data":"3925165db698ab35c9144c93aa899e90f4d970aa4386084365960d9230821774"} Nov 02 22:50:01.323518 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.323502 5109 scope.go:122] "RemoveContainer" containerID="251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d" Nov 02 22:50:01.323874 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:50:01.323848 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d\": container with ID starting with 251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d not found: ID does not exist" containerID="251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d" Nov 02 22:50:01.323986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.323926 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d"} err="failed to get container status \"251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d\": rpc error: code = NotFound desc = could not find container \"251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d\": container with ID starting with 251a2bf9dc37effd86e46850fd8f0f6b96c56d73407b481ae8585b91286d1b8d not found: ID does not exist" Nov 02 22:50:01.323986 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.323954 5109 scope.go:122] "RemoveContainer" containerID="f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b" Nov 02 22:50:01.332001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.331985 5109 scope.go:122] "RemoveContainer" containerID="f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b" Nov 02 22:50:01.332270 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:50:01.332237 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b\": container with ID starting with f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b not found: ID does not exist" containerID="f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b" Nov 02 22:50:01.332315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.332278 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b"} err="failed to get container status \"f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b\": rpc error: code = NotFound desc = could not find container \"f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b\": container with ID starting with f494bc7a4208671efbebc36418af31b936ae8622c6b46be5b5a4b7b874094f8b not found: ID does not exist" Nov 02 22:50:01.477061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.477029 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj"] Nov 02 22:50:01.480887 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.480863 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c86709-688b8w7vvj"] Nov 02 22:50:01.590488 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.590404 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6"] Nov 02 22:50:01.594166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.594141 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c86709-65d7944d8hwng6"] Nov 02 22:50:01.891214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.891133 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b742a8f-419e-4bd5-ad74-4e3a7418509a" path="/var/lib/kubelet/pods/1b742a8f-419e-4bd5-ad74-4e3a7418509a/volumes" Nov 02 22:50:01.891593 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.891552 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251ed681-3a13-455f-8d15-5c69c2916f21" path="/var/lib/kubelet/pods/251ed681-3a13-455f-8d15-5c69c2916f21/volumes" Nov 02 22:50:01.892081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:01.892062 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1b6d85-7513-4396-9e0e-de22bc36f51c" path="/var/lib/kubelet/pods/fb1b6d85-7513-4396-9e0e-de22bc36f51c/volumes" Nov 02 22:50:26.489261 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:50:26.489224 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:50:28.047554 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:28.047508 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c"] Nov 02 22:50:28.097469 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:28.097426 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9998c"] Nov 02 22:50:29.890586 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:50:29.890525 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3743acb-0bb9-4468-84b2-baaef53c97df" path="/var/lib/kubelet/pods/f3743acb-0bb9-4468-84b2-baaef53c97df/volumes" Nov 02 22:50:56.489874 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:50:56.489829 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:51:17.558978 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:17.558905 5109 scope.go:122] "RemoveContainer" containerID="33c7096f1f6ebb2de60c4faf7f5e69f2868091bcf4bf61b4f931a1b78f6ea98a" Nov 02 22:51:17.567350 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:17.567332 5109 scope.go:122] "RemoveContainer" containerID="f3947cb24ac0ed607f397e106b3025b3dca10cf9816ebb97cff5464bd4be7c27" Nov 02 22:51:25.495820 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.495777 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g"] Nov 02 22:51:25.496283 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496269 5109 cpu_manager.go:414] "containerMap: removing container" podUID="1b742a8f-419e-4bd5-ad74-4e3a7418509a" containerName="payload-processing" Nov 02 22:51:25.496333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496285 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b742a8f-419e-4bd5-ad74-4e3a7418509a" containerName="payload-processing" Nov 02 22:51:25.496333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496308 5109 cpu_manager.go:414] "containerMap: removing container" podUID="fb1b6d85-7513-4396-9e0e-de22bc36f51c" containerName="payload-pre-processing" Nov 02 22:51:25.496333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496317 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1b6d85-7513-4396-9e0e-de22bc36f51c" containerName="payload-pre-processing" Nov 02 22:51:25.496439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496343 5109 cpu_manager.go:414] "containerMap: removing container" podUID="251ed681-3a13-455f-8d15-5c69c2916f21" containerName="revoke-keys" Nov 02 22:51:25.496439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496348 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251ed681-3a13-455f-8d15-5c69c2916f21" containerName="revoke-keys" Nov 02 22:51:25.496439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496417 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="251ed681-3a13-455f-8d15-5c69c2916f21" containerName="revoke-keys" Nov 02 22:51:25.496439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496428 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b742a8f-419e-4bd5-ad74-4e3a7418509a" containerName="payload-processing" Nov 02 22:51:25.496439 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.496436 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1b6d85-7513-4396-9e0e-de22bc36f51c" containerName="payload-pre-processing" Nov 02 22:51:25.499660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.499631 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.555932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.555897 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g"] Nov 02 22:51:25.603714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.603677 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:51:25.603714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.603702 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:51:25.679617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.679539 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.679850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.679702 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp295\" (UniqueName: \"kubernetes.io/projected/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-kube-api-access-bp295\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.679850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.679775 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.781157 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.781119 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.781351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.781170 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp295\" (UniqueName: \"kubernetes.io/projected/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-kube-api-access-bp295\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.781351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.781198 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.781836 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.781811 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.783935 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.783909 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:25.844166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:25.844131 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp295\" (UniqueName: \"kubernetes.io/projected/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-kube-api-access-bp295\") pod \"maas-api-revoke-sub-452b6d7b-ttb8g\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:26.112649 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:26.112539 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:26.482062 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:51:26.482024 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63efc48_2952_4bfa_9d73_44ef0e7a20a1.slice/crio-1cdc21caaaa60624332e28e6718cfa8e494d7286a2be846e07d38eb791d82d94 WatchSource:0}: Error finding container 1cdc21caaaa60624332e28e6718cfa8e494d7286a2be846e07d38eb791d82d94: Status 404 returned error can't find the container with id 1cdc21caaaa60624332e28e6718cfa8e494d7286a2be846e07d38eb791d82d94 Nov 02 22:51:26.482971 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:26.482944 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g"] Nov 02 22:51:26.492909 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:51:26.492874 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:51:26.807460 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:26.807421 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" event={"ID":"a63efc48-2952-4bfa-9d73-44ef0e7a20a1","Type":"ContainerStarted","Data":"4e6955ad420f531de5eff062d115cc9142bb0fc457deb47ffe01705e977280eb"} Nov 02 22:51:26.807930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:26.807491 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" event={"ID":"a63efc48-2952-4bfa-9d73-44ef0e7a20a1","Type":"ContainerStarted","Data":"1cdc21caaaa60624332e28e6718cfa8e494d7286a2be846e07d38eb791d82d94"} Nov 02 22:51:26.924886 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:26.924822 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" podStartSLOduration=1.924808944 podStartE2EDuration="1.924808944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:26.92405073 +0000 UTC m=+2723.189043983" watchObservedRunningTime="2026-11-02 22:51:26.924808944 +0000 UTC m=+2723.189802162" Nov 02 22:51:27.816681 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:27.816636 5109 generic.go:356] "Generic (PLEG): container finished" podID="a63efc48-2952-4bfa-9d73-44ef0e7a20a1" containerID="4e6955ad420f531de5eff062d115cc9142bb0fc457deb47ffe01705e977280eb" exitCode=0 Nov 02 22:51:27.817165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:27.816728 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" event={"ID":"a63efc48-2952-4bfa-9d73-44ef0e7a20a1","Type":"ContainerDied","Data":"4e6955ad420f531de5eff062d115cc9142bb0fc457deb47ffe01705e977280eb"} Nov 02 22:51:29.251007 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.250732 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:29.314960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.314921 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-xks-ca\") pod \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " Nov 02 22:51:29.314960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.314982 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-openshift-ca\") pod \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " Nov 02 22:51:29.315274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.315044 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-kube-api-access-bp295\" (UniqueName: \"kubernetes.io/projected/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-kube-api-access-bp295\") pod \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\" (UID: \"a63efc48-2952-4bfa-9d73-44ef0e7a20a1\") " Nov 02 22:51:29.315374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.315346 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-openshift-ca" pod "a63efc48-2952-4bfa-9d73-44ef0e7a20a1" (UID: "a63efc48-2952-4bfa-9d73-44ef0e7a20a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:51:29.317340 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.317301 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-xks-ca" pod "a63efc48-2952-4bfa-9d73-44ef0e7a20a1" (UID: "a63efc48-2952-4bfa-9d73-44ef0e7a20a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:51:29.317480 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.317369 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-kube-api-access-bp295" pod "a63efc48-2952-4bfa-9d73-44ef0e7a20a1" (UID: "a63efc48-2952-4bfa-9d73-44ef0e7a20a1"). InnerVolumeSpecName "kube-api-access-bp295". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:51:29.416627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.416554 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp295\" (UniqueName: \"kubernetes.io/projected/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-kube-api-access-bp295\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:29.416627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.416615 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:29.416627 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.416625 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63efc48-2952-4bfa-9d73-44ef0e7a20a1-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:29.846596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.845655 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" event={"ID":"a63efc48-2952-4bfa-9d73-44ef0e7a20a1","Type":"ContainerDied","Data":"1cdc21caaaa60624332e28e6718cfa8e494d7286a2be846e07d38eb791d82d94"} Nov 02 22:51:29.846596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.845725 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cdc21caaaa60624332e28e6718cfa8e494d7286a2be846e07d38eb791d82d94" Nov 02 22:51:29.846596 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:29.845818 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g" Nov 02 22:51:50.981099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:50.981059 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr"] Nov 02 22:51:50.981841 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:50.981819 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a63efc48-2952-4bfa-9d73-44ef0e7a20a1" containerName="revoke-keys" Nov 02 22:51:50.981897 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:50.981844 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63efc48-2952-4bfa-9d73-44ef0e7a20a1" containerName="revoke-keys" Nov 02 22:51:50.981967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:50.981954 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63efc48-2952-4bfa-9d73-44ef0e7a20a1" containerName="revoke-keys" Nov 02 22:51:50.985214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:50.985198 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.040873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.040837 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr"] Nov 02 22:51:51.064868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.064829 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.065086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.064970 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfdz\" (UniqueName: \"kubernetes.io/projected/7b213c3c-c355-49b6-be6f-a944ce32c2f0-kube-api-access-qtfdz\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.065086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.065052 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.087736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.087697 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:51:51.087736 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.087727 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:51:51.165991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.165940 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.165991 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.165994 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfdz\" (UniqueName: \"kubernetes.io/projected/7b213c3c-c355-49b6-be6f-a944ce32c2f0-kube-api-access-qtfdz\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.166274 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.166036 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.166647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.166624 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.168673 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.168650 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.228001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.227961 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfdz\" (UniqueName: \"kubernetes.io/projected/7b213c3c-c355-49b6-be6f-a944ce32c2f0-kube-api-access-qtfdz\") pod \"maas-api-revoke-sub-3e31a506-f7kfr\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.298038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.297997 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:51.672424 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:51:51.672385 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b213c3c_c355_49b6_be6f_a944ce32c2f0.slice/crio-e86de84f45510eba8720aa213ef9c7e02810b6758e53d4005ca46f6c0eeaec4d WatchSource:0}: Error finding container e86de84f45510eba8720aa213ef9c7e02810b6758e53d4005ca46f6c0eeaec4d: Status 404 returned error can't find the container with id e86de84f45510eba8720aa213ef9c7e02810b6758e53d4005ca46f6c0eeaec4d Nov 02 22:51:51.674160 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.674128 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr"] Nov 02 22:51:51.961036 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.961003 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" event={"ID":"7b213c3c-c355-49b6-be6f-a944ce32c2f0","Type":"ContainerStarted","Data":"b2fb247300ec32bb0b23441c57ad0a5995f115c422a340eb59e0f0d17d4abd27"} Nov 02 22:51:51.961217 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:51.961056 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" event={"ID":"7b213c3c-c355-49b6-be6f-a944ce32c2f0","Type":"ContainerStarted","Data":"e86de84f45510eba8720aa213ef9c7e02810b6758e53d4005ca46f6c0eeaec4d"} Nov 02 22:51:52.078116 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:52.078068 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" podStartSLOduration=2.078054256 podStartE2EDuration="2.078054256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:52.076885018 +0000 UTC m=+2748.341878236" watchObservedRunningTime="2026-11-02 22:51:52.078054256 +0000 UTC m=+2748.343047476" Nov 02 22:51:52.969300 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:52.969195 5109 generic.go:356] "Generic (PLEG): container finished" podID="7b213c3c-c355-49b6-be6f-a944ce32c2f0" containerID="b2fb247300ec32bb0b23441c57ad0a5995f115c422a340eb59e0f0d17d4abd27" exitCode=0 Nov 02 22:51:52.969500 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:52.969296 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" event={"ID":"7b213c3c-c355-49b6-be6f-a944ce32c2f0","Type":"ContainerDied","Data":"b2fb247300ec32bb0b23441c57ad0a5995f115c422a340eb59e0f0d17d4abd27"} Nov 02 22:51:54.437830 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.437789 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:54.519899 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.519861 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-xks-ca\") pod \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " Nov 02 22:51:54.520109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.519955 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-openshift-ca\") pod \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " Nov 02 22:51:54.520109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.519995 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b213c3c-c355-49b6-be6f-a944ce32c2f0-kube-api-access-qtfdz\" (UniqueName: \"kubernetes.io/projected/7b213c3c-c355-49b6-be6f-a944ce32c2f0-kube-api-access-qtfdz\") pod \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\" (UID: \"7b213c3c-c355-49b6-be6f-a944ce32c2f0\") " Nov 02 22:51:54.520327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.520301 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-openshift-ca" pod "7b213c3c-c355-49b6-be6f-a944ce32c2f0" (UID: "7b213c3c-c355-49b6-be6f-a944ce32c2f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:51:54.522153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.522132 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-xks-ca" pod "7b213c3c-c355-49b6-be6f-a944ce32c2f0" (UID: "7b213c3c-c355-49b6-be6f-a944ce32c2f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:51:54.522234 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.522180 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b213c3c-c355-49b6-be6f-a944ce32c2f0-kube-api-access-qtfdz" pod "7b213c3c-c355-49b6-be6f-a944ce32c2f0" (UID: "7b213c3c-c355-49b6-be6f-a944ce32c2f0"). InnerVolumeSpecName "kube-api-access-qtfdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:51:54.621674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.621630 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.621674 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.621676 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b213c3c-c355-49b6-be6f-a944ce32c2f0-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.621888 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.621697 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfdz\" (UniqueName: \"kubernetes.io/projected/7b213c3c-c355-49b6-be6f-a944ce32c2f0-kube-api-access-qtfdz\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.993652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.993612 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" event={"ID":"7b213c3c-c355-49b6-be6f-a944ce32c2f0","Type":"ContainerDied","Data":"e86de84f45510eba8720aa213ef9c7e02810b6758e53d4005ca46f6c0eeaec4d"} Nov 02 22:51:54.993864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.993669 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e86de84f45510eba8720aa213ef9c7e02810b6758e53d4005ca46f6c0eeaec4d" Nov 02 22:51:54.993864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:51:54.993628 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr" Nov 02 22:51:56.491393 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:51:56.491290 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:52:17.592538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:17.592502 5109 scope.go:122] "RemoveContainer" containerID="9bc076bff2387949c78a517d887e81ba57a2812f4d36cbb9d701705d2505ba33" Nov 02 22:52:20.647020 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.646986 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c"] Nov 02 22:52:20.647441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.647411 5109 cpu_manager.go:414] "containerMap: removing container" podUID="7b213c3c-c355-49b6-be6f-a944ce32c2f0" containerName="revoke-keys" Nov 02 22:52:20.647441 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.647422 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b213c3c-c355-49b6-be6f-a944ce32c2f0" containerName="revoke-keys" Nov 02 22:52:20.647522 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.647494 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b213c3c-c355-49b6-be6f-a944ce32c2f0" containerName="revoke-keys" Nov 02 22:52:20.652071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.652052 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.707877 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.707842 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c"] Nov 02 22:52:20.755035 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.754998 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:52:20.755224 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.755063 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:52:20.774365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.774333 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.774498 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.774382 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2tq\" (UniqueName: \"kubernetes.io/projected/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-kube-api-access-7n2tq\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.774545 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.774522 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.876167 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.876119 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.876395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.876181 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2tq\" (UniqueName: \"kubernetes.io/projected/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-kube-api-access-7n2tq\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.876395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.876256 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.876960 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.876927 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.878687 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.878664 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.937734 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.937657 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2tq\" (UniqueName: \"kubernetes.io/projected/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-kube-api-access-7n2tq\") pod \"maas-api-revoke-sub-831829d9-q8m8c\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:20.964721 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:20.964688 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:21.339150 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:52:21.339108 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4b7b07_c1ec_439c_b1a2_90372b60fc43.slice/crio-cdd8edb441c2415327b06564db273e1690076926fa929e5ad45c269669ec2c14 WatchSource:0}: Error finding container cdd8edb441c2415327b06564db273e1690076926fa929e5ad45c269669ec2c14: Status 404 returned error can't find the container with id cdd8edb441c2415327b06564db273e1690076926fa929e5ad45c269669ec2c14 Nov 02 22:52:21.339539 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:21.339416 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c"] Nov 02 22:52:22.212466 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:22.212432 5109 generic.go:356] "Generic (PLEG): container finished" podID="bd4b7b07-c1ec-439c-b1a2-90372b60fc43" containerID="8840d3455e63ff9cea41c9e2e84658667187e900adaa3c8ffcf3615535ce0e39" exitCode=0 Nov 02 22:52:22.212922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:22.212507 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" event={"ID":"bd4b7b07-c1ec-439c-b1a2-90372b60fc43","Type":"ContainerDied","Data":"8840d3455e63ff9cea41c9e2e84658667187e900adaa3c8ffcf3615535ce0e39"} Nov 02 22:52:22.212922 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:22.212582 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" event={"ID":"bd4b7b07-c1ec-439c-b1a2-90372b60fc43","Type":"ContainerStarted","Data":"cdd8edb441c2415327b06564db273e1690076926fa929e5ad45c269669ec2c14"} Nov 02 22:52:23.359193 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.359159 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:23.402377 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.402334 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-kube-api-access-7n2tq\" (UniqueName: \"kubernetes.io/projected/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-kube-api-access-7n2tq\") pod \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " Nov 02 22:52:23.402617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.402470 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-xks-ca\") pod \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " Nov 02 22:52:23.402697 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.402631 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-openshift-ca\") pod \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\" (UID: \"bd4b7b07-c1ec-439c-b1a2-90372b60fc43\") " Nov 02 22:52:23.402979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.402954 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-openshift-ca" pod "bd4b7b07-c1ec-439c-b1a2-90372b60fc43" (UID: "bd4b7b07-c1ec-439c-b1a2-90372b60fc43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:52:23.404698 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.404678 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-xks-ca" pod "bd4b7b07-c1ec-439c-b1a2-90372b60fc43" (UID: "bd4b7b07-c1ec-439c-b1a2-90372b60fc43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:52:23.404831 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.404816 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-kube-api-access-7n2tq" pod "bd4b7b07-c1ec-439c-b1a2-90372b60fc43" (UID: "bd4b7b07-c1ec-439c-b1a2-90372b60fc43"). InnerVolumeSpecName "kube-api-access-7n2tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:52:23.504511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.504425 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:23.504511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.504457 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2tq\" (UniqueName: \"kubernetes.io/projected/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-kube-api-access-7n2tq\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:23.504511 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:23.504471 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4b7b07-c1ec-439c-b1a2-90372b60fc43-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:24.223810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:24.223726 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" Nov 02 22:52:24.223810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:24.223747 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c" event={"ID":"bd4b7b07-c1ec-439c-b1a2-90372b60fc43","Type":"ContainerDied","Data":"cdd8edb441c2415327b06564db273e1690076926fa929e5ad45c269669ec2c14"} Nov 02 22:52:24.223810 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:24.223806 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd8edb441c2415327b06564db273e1690076926fa929e5ad45c269669ec2c14" Nov 02 22:52:26.489922 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:52:26.489883 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:52:35.049833 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:35.049800 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh"] Nov 02 22:52:35.099838 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:35.099794 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kpnrh"] Nov 02 22:52:35.891401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:35.891355 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9884aa2e-c612-45aa-8004-2e1f2600d21e" path="/var/lib/kubelet/pods/9884aa2e-c612-45aa-8004-2e1f2600d21e/volumes" Nov 02 22:52:41.272374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.272332 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw"] Nov 02 22:52:41.272815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.272765 5109 cpu_manager.go:414] "containerMap: removing container" podUID="bd4b7b07-c1ec-439c-b1a2-90372b60fc43" containerName="revoke-keys" Nov 02 22:52:41.272815 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.272777 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4b7b07-c1ec-439c-b1a2-90372b60fc43" containerName="revoke-keys" Nov 02 22:52:41.272891 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.272867 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4b7b07-c1ec-439c-b1a2-90372b60fc43" containerName="revoke-keys" Nov 02 22:52:41.276198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.276175 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.342848 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.333842 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e0efc115-openshift-default-dockercfg-kgrml\"" Nov 02 22:52:41.385133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.385097 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw"] Nov 02 22:52:41.407895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.407819 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-podinfo\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.407895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.407858 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6kc\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-kube-api-access-xc6kc\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.407895 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.407887 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-data\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.408143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.408045 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-token\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.408143 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.408114 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istiod-ca-cert\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.408215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.408145 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-credential-socket\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.408215 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.408185 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-certs\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.408297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.408267 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-socket\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.408297 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.408287 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-envoy\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.509592 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509532 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-data\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.509805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509625 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-token\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.509805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509663 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istiod-ca-cert\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.509805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509685 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-credential-socket\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.509805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509725 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-certs\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.509805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509773 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-socket\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.509805 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509799 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-envoy\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.510152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509836 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-podinfo\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.510152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.509871 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6kc\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-kube-api-access-xc6kc\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.510152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.510041 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-data\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.510152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.510077 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-credential-socket\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.510152 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.510146 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-certs\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.510344 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.510187 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-socket\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.510689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.510667 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istiod-ca-cert\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.512252 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.512216 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-envoy\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.512502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.512482 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-podinfo\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.571949 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.571876 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-token\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.572105 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.572035 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6kc\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-kube-api-access-xc6kc\") pod \"e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:41.589741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:41.589707 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:42.024881 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:52:42.024846 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d947e9_4f5c_43c5_b1c6_95bab068d1d1.slice/crio-000f00adde70f575ab630c6254bebbb14e6698ffee9cc6ea3f2cae3218537236 WatchSource:0}: Error finding container 000f00adde70f575ab630c6254bebbb14e6698ffee9cc6ea3f2cae3218537236: Status 404 returned error can't find the container with id 000f00adde70f575ab630c6254bebbb14e6698ffee9cc6ea3f2cae3218537236 Nov 02 22:52:42.026103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.026080 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw"] Nov 02 22:52:42.027009 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.026973 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:52:42.027109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.027049 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:52:42.027109 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.027094 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:52:42.324060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.323975 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" event={"ID":"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1","Type":"ContainerStarted","Data":"4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37"} Nov 02 22:52:42.324060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.324034 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" event={"ID":"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1","Type":"ContainerStarted","Data":"000f00adde70f575ab630c6254bebbb14e6698ffee9cc6ea3f2cae3218537236"} Nov 02 22:52:42.445196 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.445132 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podStartSLOduration=1.445116512 podStartE2EDuration="1.445116512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:42.443779028 +0000 UTC m=+2798.708772259" watchObservedRunningTime="2026-11-02 22:52:42.445116512 +0000 UTC m=+2798.710109731" Nov 02 22:52:42.590389 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:42.590304 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:43.596201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:43.596166 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:44.335118 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:44.335081 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:44.336103 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:44.336082 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:52:56.094327 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.094286 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4"] Nov 02 22:52:56.097994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.097969 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.162866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.162817 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971a84c7-3c25-441a-8734-bc83b15695ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.163091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.162973 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2skw\" (UniqueName: \"kubernetes.io/projected/971a84c7-3c25-441a-8734-bc83b15695ce-kube-api-access-f2skw\") pod \"payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.206861 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.206821 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e0efc115\"" Nov 02 22:52:56.207135 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.206872 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4"] Nov 02 22:52:56.208172 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.206950 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e0efc115-dockercfg-fvcmh\"" Nov 02 22:52:56.255916 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.255883 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq"] Nov 02 22:52:56.259602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.259556 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.263847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.263820 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971a84c7-3c25-441a-8734-bc83b15695ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.264001 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.263931 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2skw\" (UniqueName: \"kubernetes.io/projected/971a84c7-3c25-441a-8734-bc83b15695ce-kube-api-access-f2skw\") pod \"payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.264525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.264502 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971a84c7-3c25-441a-8734-bc83b15695ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.315014 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.314980 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq"] Nov 02 22:52:56.325463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.325427 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2skw\" (UniqueName: \"kubernetes.io/projected/971a84c7-3c25-441a-8734-bc83b15695ce-kube-api-access-f2skw\") pod \"payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.364875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.364787 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ab8e01-991d-457d-892f-db7995c47d8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.364875 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.364852 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzc55\" (UniqueName: \"kubernetes.io/projected/c5ab8e01-991d-457d-892f-db7995c47d8b-kube-api-access-pzc55\") pod \"payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.412031 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.411996 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:56.465803 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.465770 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ab8e01-991d-457d-892f-db7995c47d8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.465958 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.465837 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzc55\" (UniqueName: \"kubernetes.io/projected/c5ab8e01-991d-457d-892f-db7995c47d8b-kube-api-access-pzc55\") pod \"payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.467082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.467059 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ab8e01-991d-457d-892f-db7995c47d8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.501689 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:52:56.501648 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:52:56.527192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.527154 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzc55\" (UniqueName: \"kubernetes.io/projected/c5ab8e01-991d-457d-892f-db7995c47d8b-kube-api-access-pzc55\") pod \"payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.573970 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.573927 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:56.781821 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:52:56.781792 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971a84c7_3c25_441a_8734_bc83b15695ce.slice/crio-ecb7592efe40c05b965e9a5e26a5b6e80efd2996ee5ee3c60d2a5ae1437ec10e WatchSource:0}: Error finding container ecb7592efe40c05b965e9a5e26a5b6e80efd2996ee5ee3c60d2a5ae1437ec10e: Status 404 returned error can't find the container with id ecb7592efe40c05b965e9a5e26a5b6e80efd2996ee5ee3c60d2a5ae1437ec10e Nov 02 22:52:56.782124 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.782100 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4"] Nov 02 22:52:56.946447 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:52:56.946412 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ab8e01_991d_457d_892f_db7995c47d8b.slice/crio-65697fbec48cb1187c824a1ced9cbdfc6681128f5fd3235eaee454fc29b8d4e2 WatchSource:0}: Error finding container 65697fbec48cb1187c824a1ced9cbdfc6681128f5fd3235eaee454fc29b8d4e2: Status 404 returned error can't find the container with id 65697fbec48cb1187c824a1ced9cbdfc6681128f5fd3235eaee454fc29b8d4e2 Nov 02 22:52:56.947789 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:56.947762 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq"] Nov 02 22:52:57.402207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:57.402166 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" event={"ID":"971a84c7-3c25-441a-8734-bc83b15695ce","Type":"ContainerStarted","Data":"3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0"} Nov 02 22:52:57.402786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:57.402240 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:52:57.402786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:57.402265 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" event={"ID":"971a84c7-3c25-441a-8734-bc83b15695ce","Type":"ContainerStarted","Data":"ecb7592efe40c05b965e9a5e26a5b6e80efd2996ee5ee3c60d2a5ae1437ec10e"} Nov 02 22:52:57.403967 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:57.403939 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" event={"ID":"c5ab8e01-991d-457d-892f-db7995c47d8b","Type":"ContainerStarted","Data":"4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab"} Nov 02 22:52:57.404075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:57.403988 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" event={"ID":"c5ab8e01-991d-457d-892f-db7995c47d8b","Type":"ContainerStarted","Data":"65697fbec48cb1187c824a1ced9cbdfc6681128f5fd3235eaee454fc29b8d4e2"} Nov 02 22:52:57.404075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:57.404019 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:52:57.518501 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:52:57.518447 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" podStartSLOduration=1.518431043 podStartE2EDuration="1.518431043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:57.517716392 +0000 UTC m=+2813.782709610" watchObservedRunningTime="2026-11-02 22:52:57.518431043 +0000 UTC m=+2813.783424261" Nov 02 22:53:05.049385 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:05.049283 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" podStartSLOduration=9.049268743 podStartE2EDuration="9.049268743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:57.643432514 +0000 UTC m=+2813.908425734" watchObservedRunningTime="2026-11-02 22:53:05.049268743 +0000 UTC m=+2821.314261978" Nov 02 22:53:05.050589 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:05.050544 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt"] Nov 02 22:53:05.102023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:05.101979 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-j25nt"] Nov 02 22:53:05.892774 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:05.892737 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96353aec-7d34-4e37-822c-2e5e058af60d" path="/var/lib/kubelet/pods/96353aec-7d34-4e37-822c-2e5e058af60d/volumes" Nov 02 22:53:08.412669 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.412639 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:53:08.413119 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.412713 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:53:08.487770 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.487732 5109 generic.go:356] "Generic (PLEG): container finished" podID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerID="4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37" exitCode=0 Nov 02 22:53:08.487957 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.487795 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" event={"ID":"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1","Type":"ContainerDied","Data":"4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37"} Nov 02 22:53:08.488183 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.488169 5109 scope.go:122] "RemoveContainer" containerID="4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37" Nov 02 22:53:08.489395 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.489362 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:53:08.489515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.489434 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:53:08.489515 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:08.489469 5109 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:53:09.495607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:09.495548 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" event={"ID":"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1","Type":"ContainerStarted","Data":"26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5"} Nov 02 22:53:09.590762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:09.590724 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:53:09.591004 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:09.590977 5109 patch_prober.go:29] interesting pod/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 22:53:09.591060 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:09.591038 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 22:53:10.590689 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:10.590643 5109 patch_prober.go:29] interesting pod/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 22:53:10.591086 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:10.590713 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 22:53:11.590617 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:11.590550 5109 patch_prober.go:29] interesting pod/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 22:53:11.590828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:11.590636 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 22:53:12.591030 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:12.590995 5109 patch_prober.go:29] interesting pod/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 22:53:12.591461 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:12.591047 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 22:53:13.590492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:13.590455 5109 patch_prober.go:29] interesting pod/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 22:53:13.590705 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:13.590510 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 22:53:14.343595 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:14.339682 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:53:14.590479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:14.590441 5109 patch_prober.go:29] interesting pod/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 22:53:14.590696 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:14.590500 5109 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 22:53:15.595355 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:15.595313 5109 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:53:15.596356 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:15.596331 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:53:21.147465 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.147427 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l"] Nov 02 22:53:21.152233 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.152213 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.205081 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.205047 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:53:21.254972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.254939 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zf6xm\"" Nov 02 22:53:21.257932 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.257892 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l"] Nov 02 22:53:21.267216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.267184 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4sc\" (UniqueName: \"kubernetes.io/projected/35aec918-bba9-4176-8e4c-f60cb073d86d-kube-api-access-jj4sc\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.267376 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.267237 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.267432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.267377 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.368884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.368843 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.369099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.368907 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.369099 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.368995 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4sc\" (UniqueName: \"kubernetes.io/projected/35aec918-bba9-4176-8e4c-f60cb073d86d-kube-api-access-jj4sc\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.369662 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.369636 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.371456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.371436 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.430927 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.430848 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4sc\" (UniqueName: \"kubernetes.io/projected/35aec918-bba9-4176-8e4c-f60cb073d86d-kube-api-access-jj4sc\") pod \"maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.465534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.465489 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:21.835951 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:53:21.835903 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35aec918_bba9_4176_8e4c_f60cb073d86d.slice/crio-705ec934c41b53c55f3d442c599f18312e46e26e57be8f6ed3c842b5ab45f851 WatchSource:0}: Error finding container 705ec934c41b53c55f3d442c599f18312e46e26e57be8f6ed3c842b5ab45f851: Status 404 returned error can't find the container with id 705ec934c41b53c55f3d442c599f18312e46e26e57be8f6ed3c842b5ab45f851 Nov 02 22:53:21.837153 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:21.837129 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l"] Nov 02 22:53:22.564652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:22.564606 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" event={"ID":"35aec918-bba9-4176-8e4c-f60cb073d86d","Type":"ContainerStarted","Data":"a4a8675ba9a20c34007f77781fdfc44eb5ea20e61ab795a20fa8928c478d0488"} Nov 02 22:53:22.565089 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:22.564680 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" event={"ID":"35aec918-bba9-4176-8e4c-f60cb073d86d","Type":"ContainerStarted","Data":"705ec934c41b53c55f3d442c599f18312e46e26e57be8f6ed3c842b5ab45f851"} Nov 02 22:53:22.683151 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:22.683100 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" podStartSLOduration=1.6830855009999999 podStartE2EDuration="1.683085501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:22.680899591 +0000 UTC m=+2838.945892812" watchObservedRunningTime="2026-11-02 22:53:22.683085501 +0000 UTC m=+2838.948078720" Nov 02 22:53:23.571802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:23.571767 5109 generic.go:356] "Generic (PLEG): container finished" podID="35aec918-bba9-4176-8e4c-f60cb073d86d" containerID="a4a8675ba9a20c34007f77781fdfc44eb5ea20e61ab795a20fa8928c478d0488" exitCode=0 Nov 02 22:53:23.572232 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:23.571854 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" event={"ID":"35aec918-bba9-4176-8e4c-f60cb073d86d","Type":"ContainerDied","Data":"a4a8675ba9a20c34007f77781fdfc44eb5ea20e61ab795a20fa8928c478d0488"} Nov 02 22:53:24.720786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.720763 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:24.802136 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.802098 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35aec918-bba9-4176-8e4c-f60cb073d86d-kube-api-access-jj4sc\" (UniqueName: \"kubernetes.io/projected/35aec918-bba9-4176-8e4c-f60cb073d86d-kube-api-access-jj4sc\") pod \"35aec918-bba9-4176-8e4c-f60cb073d86d\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " Nov 02 22:53:24.802338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.802144 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-xks-ca\") pod \"35aec918-bba9-4176-8e4c-f60cb073d86d\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " Nov 02 22:53:24.802338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.802174 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-openshift-ca\") pod \"35aec918-bba9-4176-8e4c-f60cb073d86d\" (UID: \"35aec918-bba9-4176-8e4c-f60cb073d86d\") " Nov 02 22:53:24.802647 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.802627 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-openshift-ca" pod "35aec918-bba9-4176-8e4c-f60cb073d86d" (UID: "35aec918-bba9-4176-8e4c-f60cb073d86d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:24.804527 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.804497 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-xks-ca" pod "35aec918-bba9-4176-8e4c-f60cb073d86d" (UID: "35aec918-bba9-4176-8e4c-f60cb073d86d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:24.804675 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.804583 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35aec918-bba9-4176-8e4c-f60cb073d86d-kube-api-access-jj4sc" pod "35aec918-bba9-4176-8e4c-f60cb073d86d" (UID: "35aec918-bba9-4176-8e4c-f60cb073d86d"). InnerVolumeSpecName "kube-api-access-jj4sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:24.903082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.902994 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:24.903082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.903024 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj4sc\" (UniqueName: \"kubernetes.io/projected/35aec918-bba9-4176-8e4c-f60cb073d86d-kube-api-access-jj4sc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:24.903082 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:24.903037 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35aec918-bba9-4176-8e4c-f60cb073d86d-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:25.584023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:25.583989 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" Nov 02 22:53:25.584213 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:25.584023 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l" event={"ID":"35aec918-bba9-4176-8e4c-f60cb073d86d","Type":"ContainerDied","Data":"705ec934c41b53c55f3d442c599f18312e46e26e57be8f6ed3c842b5ab45f851"} Nov 02 22:53:25.584213 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:25.584074 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705ec934c41b53c55f3d442c599f18312e46e26e57be8f6ed3c842b5ab45f851" Nov 02 22:53:26.492045 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:53:26.492009 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:53:32.599832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:32.599797 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq"] Nov 02 22:53:32.705298 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:32.705264 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4"] Nov 02 22:53:37.600141 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.600097 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" podUID="c5ab8e01-991d-457d-892f-db7995c47d8b" containerName="payload-processing" containerID="cri-o://4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab" gracePeriod=25 Nov 02 22:53:37.706496 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.706450 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" podUID="971a84c7-3c25-441a-8734-bc83b15695ce" containerName="payload-pre-processing" containerID="cri-o://3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0" gracePeriod=25 Nov 02 22:53:37.861839 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.861761 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:53:37.952475 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.952445 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ab8e01-991d-457d-892f-db7995c47d8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ab8e01-991d-457d-892f-db7995c47d8b-plugins-config-volume\") pod \"c5ab8e01-991d-457d-892f-db7995c47d8b\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " Nov 02 22:53:37.952601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.952499 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ab8e01-991d-457d-892f-db7995c47d8b-kube-api-access-pzc55\" (UniqueName: \"kubernetes.io/projected/c5ab8e01-991d-457d-892f-db7995c47d8b-kube-api-access-pzc55\") pod \"c5ab8e01-991d-457d-892f-db7995c47d8b\" (UID: \"c5ab8e01-991d-457d-892f-db7995c47d8b\") " Nov 02 22:53:37.953127 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.953096 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ab8e01-991d-457d-892f-db7995c47d8b-plugins-config-volume" pod "c5ab8e01-991d-457d-892f-db7995c47d8b" (UID: "c5ab8e01-991d-457d-892f-db7995c47d8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:37.955133 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.955103 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ab8e01-991d-457d-892f-db7995c47d8b-kube-api-access-pzc55" pod "c5ab8e01-991d-457d-892f-db7995c47d8b" (UID: "c5ab8e01-991d-457d-892f-db7995c47d8b"). InnerVolumeSpecName "kube-api-access-pzc55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:37.962644 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:37.962621 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:53:38.053406 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.053368 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971a84c7-3c25-441a-8734-bc83b15695ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971a84c7-3c25-441a-8734-bc83b15695ce-plugins-config-volume\") pod \"971a84c7-3c25-441a-8734-bc83b15695ce\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " Nov 02 22:53:38.053653 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.053458 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971a84c7-3c25-441a-8734-bc83b15695ce-kube-api-access-f2skw\" (UniqueName: \"kubernetes.io/projected/971a84c7-3c25-441a-8734-bc83b15695ce-kube-api-access-f2skw\") pod \"971a84c7-3c25-441a-8734-bc83b15695ce\" (UID: \"971a84c7-3c25-441a-8734-bc83b15695ce\") " Nov 02 22:53:38.053896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.053873 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971a84c7-3c25-441a-8734-bc83b15695ce-plugins-config-volume" pod "971a84c7-3c25-441a-8734-bc83b15695ce" (UID: "971a84c7-3c25-441a-8734-bc83b15695ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:38.054071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.054047 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971a84c7-3c25-441a-8734-bc83b15695ce-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:38.054071 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.054073 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ab8e01-991d-457d-892f-db7995c47d8b-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:38.054229 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.054087 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzc55\" (UniqueName: \"kubernetes.io/projected/c5ab8e01-991d-457d-892f-db7995c47d8b-kube-api-access-pzc55\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:38.055869 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.055850 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971a84c7-3c25-441a-8734-bc83b15695ce-kube-api-access-f2skw" pod "971a84c7-3c25-441a-8734-bc83b15695ce" (UID: "971a84c7-3c25-441a-8734-bc83b15695ce"). InnerVolumeSpecName "kube-api-access-f2skw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:38.155269 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.155180 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2skw\" (UniqueName: \"kubernetes.io/projected/971a84c7-3c25-441a-8734-bc83b15695ce-kube-api-access-f2skw\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:38.651126 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.651082 5109 generic.go:356] "Generic (PLEG): container finished" podID="c5ab8e01-991d-457d-892f-db7995c47d8b" containerID="4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab" exitCode=0 Nov 02 22:53:38.651643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.651151 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" Nov 02 22:53:38.651643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.651163 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" event={"ID":"c5ab8e01-991d-457d-892f-db7995c47d8b","Type":"ContainerDied","Data":"4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab"} Nov 02 22:53:38.651643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.651253 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq" event={"ID":"c5ab8e01-991d-457d-892f-db7995c47d8b","Type":"ContainerDied","Data":"65697fbec48cb1187c824a1ced9cbdfc6681128f5fd3235eaee454fc29b8d4e2"} Nov 02 22:53:38.651643 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.651285 5109 scope.go:122] "RemoveContainer" containerID="4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab" Nov 02 22:53:38.652795 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.652774 5109 generic.go:356] "Generic (PLEG): container finished" podID="971a84c7-3c25-441a-8734-bc83b15695ce" containerID="3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0" exitCode=0 Nov 02 22:53:38.652873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.652803 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" event={"ID":"971a84c7-3c25-441a-8734-bc83b15695ce","Type":"ContainerDied","Data":"3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0"} Nov 02 22:53:38.652873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.652836 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" event={"ID":"971a84c7-3c25-441a-8734-bc83b15695ce","Type":"ContainerDied","Data":"ecb7592efe40c05b965e9a5e26a5b6e80efd2996ee5ee3c60d2a5ae1437ec10e"} Nov 02 22:53:38.652873 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.652846 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4" Nov 02 22:53:38.662137 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.662059 5109 scope.go:122] "RemoveContainer" containerID="4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab" Nov 02 22:53:38.662499 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:53:38.662475 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab\": container with ID starting with 4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab not found: ID does not exist" containerID="4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab" Nov 02 22:53:38.662629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.662510 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab"} err="failed to get container status \"4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab\": rpc error: code = NotFound desc = could not find container \"4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab\": container with ID starting with 4354b2c936de1b6a2eb7c37ba21e84ffec7c035cada94e47855dd07d22da83ab not found: ID does not exist" Nov 02 22:53:38.662629 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.662533 5109 scope.go:122] "RemoveContainer" containerID="3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0" Nov 02 22:53:38.671602 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.671561 5109 scope.go:122] "RemoveContainer" containerID="3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0" Nov 02 22:53:38.671938 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:53:38.671915 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0\": container with ID starting with 3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0 not found: ID does not exist" containerID="3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0" Nov 02 22:53:38.671994 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.671947 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0"} err="failed to get container status \"3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0\": rpc error: code = NotFound desc = could not find container \"3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0\": container with ID starting with 3a394a44f197a9b1d651267e0d50f00eca586ccd6a5b39c851a351670e85efb0 not found: ID does not exist" Nov 02 22:53:38.816900 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.816860 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4"] Nov 02 22:53:38.866933 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.866839 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e0efc115-57c9b7f75f-2sff4"] Nov 02 22:53:38.977505 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.977470 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq"] Nov 02 22:53:38.984365 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:38.984333 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0efc115-598bfc4c7f-588lq"] Nov 02 22:53:39.890884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:39.890850 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971a84c7-3c25-441a-8734-bc83b15695ce" path="/var/lib/kubelet/pods/971a84c7-3c25-441a-8734-bc83b15695ce/volumes" Nov 02 22:53:39.891295 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:39.891221 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ab8e01-991d-457d-892f-db7995c47d8b" path="/var/lib/kubelet/pods/c5ab8e01-991d-457d-892f-db7995c47d8b/volumes" Nov 02 22:53:42.641391 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:42.641347 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l"] Nov 02 22:53:42.691150 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:42.691107 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0efc115-z7f7l"] Nov 02 22:53:43.890371 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:43.890340 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35aec918-bba9-4176-8e4c-f60cb073d86d" path="/var/lib/kubelet/pods/35aec918-bba9-4176-8e4c-f60cb073d86d/volumes" Nov 02 22:53:47.354948 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:47.354906 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw"] Nov 02 22:53:47.355401 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:47.355120 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" containerID="cri-o://26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5" gracePeriod=30 Nov 02 22:53:52.611955 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.611926 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:53:52.624301 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624233 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istiod-ca-cert\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624301 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624278 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-token\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-token\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624316 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-kube-api-access-xc6kc\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-kube-api-access-xc6kc\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624353 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-data\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624387 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-envoy\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624438 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-socket\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624479 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624470 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-podinfo\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624514 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-certs\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624550 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-credential-socket\") pod \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\" (UID: \"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1\") " Nov 02 22:53:52.624780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624670 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istiod-ca-cert" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:52.624780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624731 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-data" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.624780 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624759 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-socket" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.625058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624881 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-credential-socket" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.625058 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.624953 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-certs" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.625178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.625073 5109 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istiod-ca-cert\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.625178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.625093 5109 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-data\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.625178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.625108 5109 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.625178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.625119 5109 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-workload-certs\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.625178 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.625129 5109 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-credential-socket\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.626693 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.626672 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-kube-api-access-xc6kc" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "kube-api-access-xc6kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:52.626879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.626858 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-podinfo" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:53:52.627159 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.627143 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-envoy" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:52.627203 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.627173 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-token" pod "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" (UID: "a5d947e9-4f5c-43c5-b1c6-95bab068d1d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:52.725988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.725947 5109 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-token\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.725988 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.725986 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6kc\" (UniqueName: \"kubernetes.io/projected/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-kube-api-access-xc6kc\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.726222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.725997 5109 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-envoy\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.726222 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.726007 5109 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1-istio-podinfo\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:52.731492 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.731459 5109 generic.go:356] "Generic (PLEG): container finished" podID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerID="26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5" exitCode=0 Nov 02 22:53:52.731694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.731528 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" event={"ID":"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1","Type":"ContainerDied","Data":"26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5"} Nov 02 22:53:52.731694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.731539 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" Nov 02 22:53:52.731694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.731620 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw" event={"ID":"a5d947e9-4f5c-43c5-b1c6-95bab068d1d1","Type":"ContainerDied","Data":"000f00adde70f575ab630c6254bebbb14e6698ffee9cc6ea3f2cae3218537236"} Nov 02 22:53:52.731694 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.731652 5109 scope.go:122] "RemoveContainer" containerID="26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5" Nov 02 22:53:52.744624 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.744603 5109 scope.go:122] "RemoveContainer" containerID="4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37" Nov 02 22:53:52.754547 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.754529 5109 scope.go:122] "RemoveContainer" containerID="26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5" Nov 02 22:53:52.754946 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:53:52.754928 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5\": container with ID starting with 26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5 not found: ID does not exist" containerID="26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5" Nov 02 22:53:52.754998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.754957 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5"} err="failed to get container status \"26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5\": rpc error: code = NotFound desc = could not find container \"26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5\": container with ID starting with 26b2ee9289d8ba39ee2d6cf5a6b810cdaa951ebb6780993769febbe3c47f1be5 not found: ID does not exist" Nov 02 22:53:52.754998 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.754978 5109 scope.go:122] "RemoveContainer" containerID="4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37" Nov 02 22:53:52.755221 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:53:52.755203 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37\": container with ID starting with 4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37 not found: ID does not exist" containerID="4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37" Nov 02 22:53:52.755281 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.755227 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37"} err="failed to get container status \"4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37\": rpc error: code = NotFound desc = could not find container \"4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37\": container with ID starting with 4ec2924a32041cc614fe0dbe1929a6c944c407e1bb3fe216e0f4f0113c639f37 not found: ID does not exist" Nov 02 22:53:52.895332 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.895247 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw"] Nov 02 22:53:52.901961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:52.901933 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e0efc115-openshift-default-5dc6d5b485-rd2nw"] Nov 02 22:53:53.890982 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:53:53.890947 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" path="/var/lib/kubelet/pods/a5d947e9-4f5c-43c5-b1c6-95bab068d1d1/volumes" Nov 02 22:53:56.491321 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:53:56.491283 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:54:05.839308 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.839265 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn"] Nov 02 22:54:05.840380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840351 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" Nov 02 22:54:05.840525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840381 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" Nov 02 22:54:05.840525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840423 5109 cpu_manager.go:414] "containerMap: removing container" podUID="971a84c7-3c25-441a-8734-bc83b15695ce" containerName="payload-pre-processing" Nov 02 22:54:05.840525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840431 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971a84c7-3c25-441a-8734-bc83b15695ce" containerName="payload-pre-processing" Nov 02 22:54:05.840525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840484 5109 cpu_manager.go:414] "containerMap: removing container" podUID="35aec918-bba9-4176-8e4c-f60cb073d86d" containerName="revoke-keys" Nov 02 22:54:05.840525 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840492 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35aec918-bba9-4176-8e4c-f60cb073d86d" containerName="revoke-keys" Nov 02 22:54:05.840818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840527 5109 cpu_manager.go:414] "containerMap: removing container" podUID="c5ab8e01-991d-457d-892f-db7995c47d8b" containerName="payload-processing" Nov 02 22:54:05.840818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840536 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ab8e01-991d-457d-892f-db7995c47d8b" containerName="payload-processing" Nov 02 22:54:05.840818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840766 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="35aec918-bba9-4176-8e4c-f60cb073d86d" containerName="revoke-keys" Nov 02 22:54:05.840818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840793 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ab8e01-991d-457d-892f-db7995c47d8b" containerName="payload-processing" Nov 02 22:54:05.840818 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840805 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" Nov 02 22:54:05.841075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840823 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" Nov 02 22:54:05.841075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.840842 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="971a84c7-3c25-441a-8734-bc83b15695ce" containerName="payload-pre-processing" Nov 02 22:54:05.845120 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.845096 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:05.897965 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.897935 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn"] Nov 02 22:54:05.961598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.949808 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-9d4413\"" Nov 02 22:54:05.961598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.950589 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-9d4413-dockercfg-96gs8\"" Nov 02 22:54:05.961598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.957491 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f1af1-ce56-49e7-8750-de30d05894ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:05.961598 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.957681 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glvkh\" (UniqueName: \"kubernetes.io/projected/216f1af1-ce56-49e7-8750-de30d05894ea-kube-api-access-glvkh\") pod \"payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:05.999374 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.999339 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c"] Nov 02 22:54:05.999987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.999966 5109 cpu_manager.go:414] "containerMap: removing container" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" Nov 02 22:54:05.999987 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:05.999988 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d947e9-4f5c-43c5-b1c6-95bab068d1d1" containerName="istio-proxy" Nov 02 22:54:06.003487 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.003466 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.058322 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.058283 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5424\" (UniqueName: \"kubernetes.io/projected/aea6daf1-ebce-4df5-bc8e-c47d24013c26-kube-api-access-r5424\") pod \"payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.058546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.058428 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea6daf1-ebce-4df5-bc8e-c47d24013c26-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.058546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.058480 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glvkh\" (UniqueName: \"kubernetes.io/projected/216f1af1-ce56-49e7-8750-de30d05894ea-kube-api-access-glvkh\") pod \"payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:06.058717 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.058549 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f1af1-ce56-49e7-8750-de30d05894ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:06.059455 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.059427 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f1af1-ce56-49e7-8750-de30d05894ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:06.059828 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.059808 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c"] Nov 02 22:54:06.120979 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.120899 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glvkh\" (UniqueName: \"kubernetes.io/projected/216f1af1-ce56-49e7-8750-de30d05894ea-kube-api-access-glvkh\") pod \"payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:06.159333 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.159288 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5424\" (UniqueName: \"kubernetes.io/projected/aea6daf1-ebce-4df5-bc8e-c47d24013c26-kube-api-access-r5424\") pod \"payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.159559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.159356 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea6daf1-ebce-4df5-bc8e-c47d24013c26-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.159559 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.159501 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:06.160212 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.160185 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea6daf1-ebce-4df5-bc8e-c47d24013c26-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.233294 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.233258 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5424\" (UniqueName: \"kubernetes.io/projected/aea6daf1-ebce-4df5-bc8e-c47d24013c26-kube-api-access-r5424\") pod \"payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.319015 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.318979 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.529540 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:54:06.529501 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f1af1_ce56_49e7_8750_de30d05894ea.slice/crio-c5eab7e8f56a1405b0581be24f756158417bde790bc7bd79be14e4093dcfbc00 WatchSource:0}: Error finding container c5eab7e8f56a1405b0581be24f756158417bde790bc7bd79be14e4093dcfbc00: Status 404 returned error can't find the container with id c5eab7e8f56a1405b0581be24f756158417bde790bc7bd79be14e4093dcfbc00 Nov 02 22:54:06.530204 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.530178 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn"] Nov 02 22:54:06.688078 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:54:06.688045 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea6daf1_ebce_4df5_bc8e_c47d24013c26.slice/crio-ff563b35ee05444cf66de10937c5a3769216b2c50fe185bbf1da0dbdc5fa3f65 WatchSource:0}: Error finding container ff563b35ee05444cf66de10937c5a3769216b2c50fe185bbf1da0dbdc5fa3f65: Status 404 returned error can't find the container with id ff563b35ee05444cf66de10937c5a3769216b2c50fe185bbf1da0dbdc5fa3f65 Nov 02 22:54:06.689549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.689523 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c"] Nov 02 22:54:06.806502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.806458 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" event={"ID":"aea6daf1-ebce-4df5-bc8e-c47d24013c26","Type":"ContainerStarted","Data":"d49c318f631fb0e024d968c516b2d091c1c79a586b518d0bd730eeacff88f3a2"} Nov 02 22:54:06.806714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.806520 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" event={"ID":"aea6daf1-ebce-4df5-bc8e-c47d24013c26","Type":"ContainerStarted","Data":"ff563b35ee05444cf66de10937c5a3769216b2c50fe185bbf1da0dbdc5fa3f65"} Nov 02 22:54:06.806714 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.806549 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:06.808163 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.808134 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" event={"ID":"216f1af1-ce56-49e7-8750-de30d05894ea","Type":"ContainerStarted","Data":"688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362"} Nov 02 22:54:06.808306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.808185 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" event={"ID":"216f1af1-ce56-49e7-8750-de30d05894ea","Type":"ContainerStarted","Data":"c5eab7e8f56a1405b0581be24f756158417bde790bc7bd79be14e4093dcfbc00"} Nov 02 22:54:06.808306 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.808206 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:06.923665 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:06.923618 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" podStartSLOduration=1.923604903 podStartE2EDuration="1.923604903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:06.922925848 +0000 UTC m=+2883.187919067" watchObservedRunningTime="2026-11-02 22:54:06.923604903 +0000 UTC m=+2883.188598116" Nov 02 22:54:07.042091 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:07.042022 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" podStartSLOduration=2.041998897 podStartE2EDuration="2.041998897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:07.041189697 +0000 UTC m=+2883.306182940" watchObservedRunningTime="2026-11-02 22:54:07.041998897 +0000 UTC m=+2883.306992119" Nov 02 22:54:17.621871 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:17.621771 5109 scope.go:122] "RemoveContainer" containerID="95b1ca53881159d8a622fc98637ec053107b4de2afcc6636654a9f0aa9e57e3e" Nov 02 22:54:17.631162 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:17.631143 5109 scope.go:122] "RemoveContainer" containerID="d5c8b600237adb84142c3088290f0fa89bc131a08f4c77c7aafc0e3ef854ee38" Nov 02 22:54:17.815523 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:17.815493 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:54:17.815737 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:17.815596 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:54:26.489759 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:54:26.489722 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:54:36.048884 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:36.048845 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k"] Nov 02 22:54:36.098969 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:36.098934 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-337c8ec2-69q4k"] Nov 02 22:54:37.890847 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:54:37.890811 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acd53a4-9335-41c6-9c64-03a2e4d1c1da" path="/var/lib/kubelet/pods/2acd53a4-9335-41c6-9c64-03a2e4d1c1da/volumes" Nov 02 22:54:56.490188 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:54:56.490142 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:55:26.490932 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:55:26.490887 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:55:56.491237 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:55:56.491187 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:55:58.850028 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.849989 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n"] Nov 02 22:55:58.853707 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.853685 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:58.896591 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.896528 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q54l\" (UniqueName: \"kubernetes.io/projected/2792afa3-71ab-4bec-b61c-685671ba09c3-kube-api-access-4q54l\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:58.896783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.896654 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:58.896783 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.896695 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:58.906848 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.906817 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:55:58.956666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.956621 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zf6xm\"" Nov 02 22:55:58.958589 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.958543 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n"] Nov 02 22:55:58.998057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.998017 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:58.998276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.998065 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:58.998276 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.998224 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q54l\" (UniqueName: \"kubernetes.io/projected/2792afa3-71ab-4bec-b61c-685671ba09c3-kube-api-access-4q54l\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:58.998832 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:58.998805 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:59.000546 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:59.000526 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:59.059863 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:59.059831 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q54l\" (UniqueName: \"kubernetes.io/projected/2792afa3-71ab-4bec-b61c-685671ba09c3-kube-api-access-4q54l\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:59.167829 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:59.167730 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:55:59.539763 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:55:59.539726 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2792afa3_71ab_4bec_b61c_685671ba09c3.slice/crio-18a537771c9ec70ba740b54dab185e59f2d4c411f94fef93bf0e09cf861438be WatchSource:0}: Error finding container 18a537771c9ec70ba740b54dab185e59f2d4c411f94fef93bf0e09cf861438be: Status 404 returned error can't find the container with id 18a537771c9ec70ba740b54dab185e59f2d4c411f94fef93bf0e09cf861438be Nov 02 22:55:59.540018 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:55:59.539995 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n"] Nov 02 22:56:00.383063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:00.383016 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" event={"ID":"2792afa3-71ab-4bec-b61c-685671ba09c3","Type":"ContainerStarted","Data":"37c765dac66f3fed6290fb053b5e721e60f38dcda50fa5ccd588a0d6aa9ea8d0"} Nov 02 22:56:00.383506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:00.383076 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" event={"ID":"2792afa3-71ab-4bec-b61c-685671ba09c3","Type":"ContainerStarted","Data":"18a537771c9ec70ba740b54dab185e59f2d4c411f94fef93bf0e09cf861438be"} Nov 02 22:56:00.501746 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:00.501693 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" podStartSLOduration=2.501677389 podStartE2EDuration="2.501677389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:00.500201333 +0000 UTC m=+2996.765194552" watchObservedRunningTime="2026-11-02 22:56:00.501677389 +0000 UTC m=+2996.766670607" Nov 02 22:56:01.389972 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:01.389927 5109 generic.go:356] "Generic (PLEG): container finished" podID="2792afa3-71ab-4bec-b61c-685671ba09c3" containerID="37c765dac66f3fed6290fb053b5e721e60f38dcda50fa5ccd588a0d6aa9ea8d0" exitCode=0 Nov 02 22:56:01.390409 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:01.390010 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" event={"ID":"2792afa3-71ab-4bec-b61c-685671ba09c3","Type":"ContainerDied","Data":"37c765dac66f3fed6290fb053b5e721e60f38dcda50fa5ccd588a0d6aa9ea8d0"} Nov 02 22:56:02.537057 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.537029 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:56:02.638260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.638214 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-xks-ca\") pod \"2792afa3-71ab-4bec-b61c-685671ba09c3\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " Nov 02 22:56:02.638260 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.638264 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2792afa3-71ab-4bec-b61c-685671ba09c3-kube-api-access-4q54l\" (UniqueName: \"kubernetes.io/projected/2792afa3-71ab-4bec-b61c-685671ba09c3-kube-api-access-4q54l\") pod \"2792afa3-71ab-4bec-b61c-685671ba09c3\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " Nov 02 22:56:02.638494 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.638352 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-openshift-ca\") pod \"2792afa3-71ab-4bec-b61c-685671ba09c3\" (UID: \"2792afa3-71ab-4bec-b61c-685671ba09c3\") " Nov 02 22:56:02.638811 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.638777 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-openshift-ca" pod "2792afa3-71ab-4bec-b61c-685671ba09c3" (UID: "2792afa3-71ab-4bec-b61c-685671ba09c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:02.640635 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.640608 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-xks-ca" pod "2792afa3-71ab-4bec-b61c-685671ba09c3" (UID: "2792afa3-71ab-4bec-b61c-685671ba09c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:02.640762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.640736 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2792afa3-71ab-4bec-b61c-685671ba09c3-kube-api-access-4q54l" pod "2792afa3-71ab-4bec-b61c-685671ba09c3" (UID: "2792afa3-71ab-4bec-b61c-685671ba09c3"). InnerVolumeSpecName "kube-api-access-4q54l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:02.739764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.739723 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:02.739764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.739757 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q54l\" (UniqueName: \"kubernetes.io/projected/2792afa3-71ab-4bec-b61c-685671ba09c3-kube-api-access-4q54l\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:02.739764 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:02.739770 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2792afa3-71ab-4bec-b61c-685671ba09c3-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:03.402227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:03.402187 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" Nov 02 22:56:03.402227 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:03.402213 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n" event={"ID":"2792afa3-71ab-4bec-b61c-685671ba09c3","Type":"ContainerDied","Data":"18a537771c9ec70ba740b54dab185e59f2d4c411f94fef93bf0e09cf861438be"} Nov 02 22:56:03.402468 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:03.402266 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18a537771c9ec70ba740b54dab185e59f2d4c411f94fef93bf0e09cf861438be" Nov 02 22:56:10.311549 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:10.311499 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c"] Nov 02 22:56:10.414741 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:10.414703 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn"] Nov 02 22:56:15.312167 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.312088 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" podUID="aea6daf1-ebce-4df5-bc8e-c47d24013c26" containerName="payload-processing" containerID="cri-o://d49c318f631fb0e024d968c516b2d091c1c79a586b518d0bd730eeacff88f3a2" gracePeriod=25 Nov 02 22:56:15.415656 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.415610 5109 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" podUID="216f1af1-ce56-49e7-8750-de30d05894ea" containerName="payload-pre-processing" containerID="cri-o://688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362" gracePeriod=25 Nov 02 22:56:15.470691 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.470647 5109 generic.go:356] "Generic (PLEG): container finished" podID="aea6daf1-ebce-4df5-bc8e-c47d24013c26" containerID="d49c318f631fb0e024d968c516b2d091c1c79a586b518d0bd730eeacff88f3a2" exitCode=0 Nov 02 22:56:15.470896 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.470689 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" event={"ID":"aea6daf1-ebce-4df5-bc8e-c47d24013c26","Type":"ContainerDied","Data":"d49c318f631fb0e024d968c516b2d091c1c79a586b518d0bd730eeacff88f3a2"} Nov 02 22:56:15.571603 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.571511 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:56:15.672743 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.672710 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea6daf1-ebce-4df5-bc8e-c47d24013c26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea6daf1-ebce-4df5-bc8e-c47d24013c26-plugins-config-volume\") pod \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " Nov 02 22:56:15.672930 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.672760 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea6daf1-ebce-4df5-bc8e-c47d24013c26-kube-api-access-r5424\" (UniqueName: \"kubernetes.io/projected/aea6daf1-ebce-4df5-bc8e-c47d24013c26-kube-api-access-r5424\") pod \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\" (UID: \"aea6daf1-ebce-4df5-bc8e-c47d24013c26\") " Nov 02 22:56:15.673169 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.673143 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea6daf1-ebce-4df5-bc8e-c47d24013c26-plugins-config-volume" pod "aea6daf1-ebce-4df5-bc8e-c47d24013c26" (UID: "aea6daf1-ebce-4df5-bc8e-c47d24013c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:15.675192 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.675167 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea6daf1-ebce-4df5-bc8e-c47d24013c26-kube-api-access-r5424" pod "aea6daf1-ebce-4df5-bc8e-c47d24013c26" (UID: "aea6daf1-ebce-4df5-bc8e-c47d24013c26"). InnerVolumeSpecName "kube-api-access-r5424". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:15.678850 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.678829 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:56:15.774243 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.774204 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216f1af1-ce56-49e7-8750-de30d05894ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f1af1-ce56-49e7-8750-de30d05894ea-plugins-config-volume\") pod \"216f1af1-ce56-49e7-8750-de30d05894ea\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " Nov 02 22:56:15.774458 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.774292 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216f1af1-ce56-49e7-8750-de30d05894ea-kube-api-access-glvkh\" (UniqueName: \"kubernetes.io/projected/216f1af1-ce56-49e7-8750-de30d05894ea-kube-api-access-glvkh\") pod \"216f1af1-ce56-49e7-8750-de30d05894ea\" (UID: \"216f1af1-ce56-49e7-8750-de30d05894ea\") " Nov 02 22:56:15.774542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.774530 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea6daf1-ebce-4df5-bc8e-c47d24013c26-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:15.774609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.774545 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5424\" (UniqueName: \"kubernetes.io/projected/aea6daf1-ebce-4df5-bc8e-c47d24013c26-kube-api-access-r5424\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:15.774666 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.774639 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216f1af1-ce56-49e7-8750-de30d05894ea-plugins-config-volume" pod "216f1af1-ce56-49e7-8750-de30d05894ea" (UID: "216f1af1-ce56-49e7-8750-de30d05894ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:15.776472 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.776452 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f1af1-ce56-49e7-8750-de30d05894ea-kube-api-access-glvkh" pod "216f1af1-ce56-49e7-8750-de30d05894ea" (UID: "216f1af1-ce56-49e7-8750-de30d05894ea"). InnerVolumeSpecName "kube-api-access-glvkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:15.875363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.875268 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glvkh\" (UniqueName: \"kubernetes.io/projected/216f1af1-ce56-49e7-8750-de30d05894ea-kube-api-access-glvkh\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:15.875363 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:15.875301 5109 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f1af1-ce56-49e7-8750-de30d05894ea-plugins-config-volume\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:16.477063 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.477029 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" Nov 02 22:56:16.477611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.477053 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c" event={"ID":"aea6daf1-ebce-4df5-bc8e-c47d24013c26","Type":"ContainerDied","Data":"ff563b35ee05444cf66de10937c5a3769216b2c50fe185bbf1da0dbdc5fa3f65"} Nov 02 22:56:16.477611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.477134 5109 scope.go:122] "RemoveContainer" containerID="d49c318f631fb0e024d968c516b2d091c1c79a586b518d0bd730eeacff88f3a2" Nov 02 22:56:16.478463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.478376 5109 generic.go:356] "Generic (PLEG): container finished" podID="216f1af1-ce56-49e7-8750-de30d05894ea" containerID="688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362" exitCode=0 Nov 02 22:56:16.478463 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.478412 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" event={"ID":"216f1af1-ce56-49e7-8750-de30d05894ea","Type":"ContainerDied","Data":"688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362"} Nov 02 22:56:16.478633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.478461 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" Nov 02 22:56:16.478633 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.478466 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn" event={"ID":"216f1af1-ce56-49e7-8750-de30d05894ea","Type":"ContainerDied","Data":"c5eab7e8f56a1405b0581be24f756158417bde790bc7bd79be14e4093dcfbc00"} Nov 02 22:56:16.486538 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.486518 5109 scope.go:122] "RemoveContainer" containerID="688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362" Nov 02 22:56:16.495062 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.495040 5109 scope.go:122] "RemoveContainer" containerID="688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362" Nov 02 22:56:16.495352 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:56:16.495330 5109 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362\": container with ID starting with 688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362 not found: ID does not exist" containerID="688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362" Nov 02 22:56:16.495436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.495360 5109 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362"} err="failed to get container status \"688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362\": rpc error: code = NotFound desc = could not find container \"688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362\": container with ID starting with 688302de27602f4e7ae88dfd1c5527f35aa3d9eb286e1b8069248d9875d69362 not found: ID does not exist" Nov 02 22:56:16.642929 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.642844 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn"] Nov 02 22:56:16.649791 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.649755 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-9d4413-8bcd6659f-tv4vn"] Nov 02 22:56:16.803652 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.803615 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c"] Nov 02 22:56:16.807953 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:16.807927 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-9d4413-579f464bfc-qnc4c"] Nov 02 22:56:17.679611 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:17.679553 5109 scope.go:122] "RemoveContainer" containerID="a5bce91f69f45c3d47be469f3f53e53f2cbc589442f8f6d89e3300f0f1110b2c" Nov 02 22:56:17.688560 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:17.688539 5109 scope.go:122] "RemoveContainer" containerID="653c1a9937869e84090f1cd3176ccc963a8f280c0673910977e473d2b853140f" Nov 02 22:56:17.891868 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:17.891827 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216f1af1-ce56-49e7-8750-de30d05894ea" path="/var/lib/kubelet/pods/216f1af1-ce56-49e7-8750-de30d05894ea/volumes" Nov 02 22:56:17.892279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:17.892259 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea6daf1-ebce-4df5-bc8e-c47d24013c26" path="/var/lib/kubelet/pods/aea6daf1-ebce-4df5-bc8e-c47d24013c26/volumes" Nov 02 22:56:21.498310 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:21.498270 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n"] Nov 02 22:56:21.548961 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:21.548921 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9d4413-ghg8n"] Nov 02 22:56:21.890699 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:21.890665 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2792afa3-71ab-4bec-b61c-685671ba09c3" path="/var/lib/kubelet/pods/2792afa3-71ab-4bec-b61c-685671ba09c3/volumes" Nov 02 22:56:26.490622 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:56:26.490587 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:56:29.050216 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:29.050179 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g"] Nov 02 22:56:29.100309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:29.100265 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ttb8g"] Nov 02 22:56:29.890786 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:29.890749 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63efc48-2952-4bfa-9d73-44ef0e7a20a1" path="/var/lib/kubelet/pods/a63efc48-2952-4bfa-9d73-44ef0e7a20a1/volumes" Nov 02 22:56:55.047747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:55.047705 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr"] Nov 02 22:56:55.097664 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:55.097622 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f7kfr"] Nov 02 22:56:55.892409 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:56:55.892370 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b213c3c-c355-49b6-be6f-a944ce32c2f0" path="/var/lib/kubelet/pods/7b213c3c-c355-49b6-be6f-a944ce32c2f0/volumes" Nov 02 22:56:56.490557 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:56:56.490516 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:57:06.066996 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.066958 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8"] Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067403 5109 cpu_manager.go:414] "containerMap: removing container" podUID="216f1af1-ce56-49e7-8750-de30d05894ea" containerName="payload-pre-processing" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067415 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216f1af1-ce56-49e7-8750-de30d05894ea" containerName="payload-pre-processing" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067426 5109 cpu_manager.go:414] "containerMap: removing container" podUID="2792afa3-71ab-4bec-b61c-685671ba09c3" containerName="revoke-keys" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067432 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2792afa3-71ab-4bec-b61c-685671ba09c3" containerName="revoke-keys" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067444 5109 cpu_manager.go:414] "containerMap: removing container" podUID="aea6daf1-ebce-4df5-bc8e-c47d24013c26" containerName="payload-processing" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067449 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea6daf1-ebce-4df5-bc8e-c47d24013c26" containerName="payload-processing" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067506 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="216f1af1-ce56-49e7-8750-de30d05894ea" containerName="payload-pre-processing" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067516 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="2792afa3-71ab-4bec-b61c-685671ba09c3" containerName="revoke-keys" Nov 02 22:57:06.067534 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.067523 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea6daf1-ebce-4df5-bc8e-c47d24013c26" containerName="payload-processing" Nov 02 22:57:06.070984 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.070966 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.126542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.126512 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8"] Nov 02 22:57:06.174436 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.174404 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:57:06.174667 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.174501 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zf6xm\"" Nov 02 22:57:06.196034 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.195991 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.196200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.196052 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.196200 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.196181 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9mn\" (UniqueName: \"kubernetes.io/projected/217f9ab7-a5f6-471f-a8fa-d346c6717509-kube-api-access-jk9mn\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.296976 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.296936 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.297166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.296994 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.297166 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.297037 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9mn\" (UniqueName: \"kubernetes.io/projected/217f9ab7-a5f6-471f-a8fa-d346c6717509-kube-api-access-jk9mn\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.297699 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.297669 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.299450 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.299431 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.359174 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.359097 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9mn\" (UniqueName: \"kubernetes.io/projected/217f9ab7-a5f6-471f-a8fa-d346c6717509-kube-api-access-jk9mn\") pod \"maas-api-revoke-sub-f03629db-zmtw8\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.384093 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.384055 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:06.753950 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:57:06.753913 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217f9ab7_a5f6_471f_a8fa_d346c6717509.slice/crio-a21b94fd9c97a32c46800eb95110f6dd12d51ff799f5d71f5fb84a530dea5a68 WatchSource:0}: Error finding container a21b94fd9c97a32c46800eb95110f6dd12d51ff799f5d71f5fb84a530dea5a68: Status 404 returned error can't find the container with id a21b94fd9c97a32c46800eb95110f6dd12d51ff799f5d71f5fb84a530dea5a68 Nov 02 22:57:06.755198 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.755170 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8"] Nov 02 22:57:06.786077 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:06.785829 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" event={"ID":"217f9ab7-a5f6-471f-a8fa-d346c6717509","Type":"ContainerStarted","Data":"a21b94fd9c97a32c46800eb95110f6dd12d51ff799f5d71f5fb84a530dea5a68"} Nov 02 22:57:07.792840 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:07.792808 5109 generic.go:356] "Generic (PLEG): container finished" podID="217f9ab7-a5f6-471f-a8fa-d346c6717509" containerID="575d336e574c72a301a183f02504420f9e3a26024a7c053256e3911227fb1194" exitCode=0 Nov 02 22:57:07.793279 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:07.792902 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" event={"ID":"217f9ab7-a5f6-471f-a8fa-d346c6717509","Type":"ContainerDied","Data":"575d336e574c72a301a183f02504420f9e3a26024a7c053256e3911227fb1194"} Nov 02 22:57:08.937205 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:08.937177 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:09.024398 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.024362 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217f9ab7-a5f6-471f-a8fa-d346c6717509-kube-api-access-jk9mn\" (UniqueName: \"kubernetes.io/projected/217f9ab7-a5f6-471f-a8fa-d346c6717509-kube-api-access-jk9mn\") pod \"217f9ab7-a5f6-471f-a8fa-d346c6717509\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " Nov 02 22:57:09.024636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.024505 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-openshift-ca\") pod \"217f9ab7-a5f6-471f-a8fa-d346c6717509\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " Nov 02 22:57:09.024636 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.024555 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-xks-ca\") pod \"217f9ab7-a5f6-471f-a8fa-d346c6717509\" (UID: \"217f9ab7-a5f6-471f-a8fa-d346c6717509\") " Nov 02 22:57:09.024983 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.024952 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-openshift-ca" pod "217f9ab7-a5f6-471f-a8fa-d346c6717509" (UID: "217f9ab7-a5f6-471f-a8fa-d346c6717509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:57:09.026722 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.026699 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217f9ab7-a5f6-471f-a8fa-d346c6717509-kube-api-access-jk9mn" pod "217f9ab7-a5f6-471f-a8fa-d346c6717509" (UID: "217f9ab7-a5f6-471f-a8fa-d346c6717509"). InnerVolumeSpecName "kube-api-access-jk9mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:57:09.026864 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.026837 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-xks-ca" pod "217f9ab7-a5f6-471f-a8fa-d346c6717509" (UID: "217f9ab7-a5f6-471f-a8fa-d346c6717509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:09.126102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.126012 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:09.126102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.126045 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk9mn\" (UniqueName: \"kubernetes.io/projected/217f9ab7-a5f6-471f-a8fa-d346c6717509-kube-api-access-jk9mn\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:09.126102 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.126055 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217f9ab7-a5f6-471f-a8fa-d346c6717509-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:09.804323 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.804291 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" Nov 02 22:57:09.804551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.804329 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8" event={"ID":"217f9ab7-a5f6-471f-a8fa-d346c6717509","Type":"ContainerDied","Data":"a21b94fd9c97a32c46800eb95110f6dd12d51ff799f5d71f5fb84a530dea5a68"} Nov 02 22:57:09.804551 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:09.804377 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a21b94fd9c97a32c46800eb95110f6dd12d51ff799f5d71f5fb84a530dea5a68" Nov 02 22:57:24.050504 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:24.050406 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c"] Nov 02 22:57:24.099902 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:24.099857 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-831829d9-q8m8c"] Nov 02 22:57:25.898502 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:25.898463 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4b7b07-c1ec-439c-b1a2-90372b60fc43" path="/var/lib/kubelet/pods/bd4b7b07-c1ec-439c-b1a2-90372b60fc43/volumes" Nov 02 22:57:26.504617 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:57:26.504548 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:57:35.237415 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.237378 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q"] Nov 02 22:57:35.237923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.237794 5109 cpu_manager.go:414] "containerMap: removing container" podUID="217f9ab7-a5f6-471f-a8fa-d346c6717509" containerName="revoke-keys" Nov 02 22:57:35.237923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.237806 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="217f9ab7-a5f6-471f-a8fa-d346c6717509" containerName="revoke-keys" Nov 02 22:57:35.237923 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.237890 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="217f9ab7-a5f6-471f-a8fa-d346c6717509" containerName="revoke-keys" Nov 02 22:57:35.240879 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.240863 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.305317 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.305279 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrhm\" (UniqueName: \"kubernetes.io/projected/849116bf-fbe4-4dd0-9187-92817cee17fb-kube-api-access-sfrhm\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.305490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.305334 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.305490 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.305470 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.344802 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.344762 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:57:35.345006 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.344825 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zf6xm\"" Nov 02 22:57:35.346506 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.346480 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q"] Nov 02 22:57:35.406311 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.406250 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.406503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.406387 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.406503 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.406486 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrhm\" (UniqueName: \"kubernetes.io/projected/849116bf-fbe4-4dd0-9187-92817cee17fb-kube-api-access-sfrhm\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.407214 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.407136 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.409061 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.409032 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.468380 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.468350 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrhm\" (UniqueName: \"kubernetes.io/projected/849116bf-fbe4-4dd0-9187-92817cee17fb-kube-api-access-sfrhm\") pod \"maas-api-revoke-sub-0165d7d2-kxl2q\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.554231 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.554178 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:35.926146 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.926116 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q"] Nov 02 22:57:35.926327 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:57:35.926187 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849116bf_fbe4_4dd0_9187_92817cee17fb.slice/crio-05001abda32a2476f5ad683e8297383e3dcf0a6c6bd5dd21d06f56ad03ee510a WatchSource:0}: Error finding container 05001abda32a2476f5ad683e8297383e3dcf0a6c6bd5dd21d06f56ad03ee510a: Status 404 returned error can't find the container with id 05001abda32a2476f5ad683e8297383e3dcf0a6c6bd5dd21d06f56ad03ee510a Nov 02 22:57:35.956544 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:35.956506 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" event={"ID":"849116bf-fbe4-4dd0-9187-92817cee17fb","Type":"ContainerStarted","Data":"05001abda32a2476f5ad683e8297383e3dcf0a6c6bd5dd21d06f56ad03ee510a"} Nov 02 22:57:36.963711 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:36.963672 5109 generic.go:356] "Generic (PLEG): container finished" podID="849116bf-fbe4-4dd0-9187-92817cee17fb" containerID="04445bcfb8f11fdd223a7dd516ca3f4a739f303a277a51c4b6a59c41f4010fc3" exitCode=0 Nov 02 22:57:36.964106 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:36.963754 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" event={"ID":"849116bf-fbe4-4dd0-9187-92817cee17fb","Type":"ContainerDied","Data":"04445bcfb8f11fdd223a7dd516ca3f4a739f303a277a51c4b6a59c41f4010fc3"} Nov 02 22:57:38.108427 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.108395 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:38.233023 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.232935 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-openshift-ca\") pod \"849116bf-fbe4-4dd0-9187-92817cee17fb\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " Nov 02 22:57:38.233207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.233024 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849116bf-fbe4-4dd0-9187-92817cee17fb-kube-api-access-sfrhm\" (UniqueName: \"kubernetes.io/projected/849116bf-fbe4-4dd0-9187-92817cee17fb-kube-api-access-sfrhm\") pod \"849116bf-fbe4-4dd0-9187-92817cee17fb\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " Nov 02 22:57:38.233207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.233097 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-xks-ca\") pod \"849116bf-fbe4-4dd0-9187-92817cee17fb\" (UID: \"849116bf-fbe4-4dd0-9187-92817cee17fb\") " Nov 02 22:57:38.234315 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.234279 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-openshift-ca" pod "849116bf-fbe4-4dd0-9187-92817cee17fb" (UID: "849116bf-fbe4-4dd0-9187-92817cee17fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:57:38.237432 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.237405 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-xks-ca" pod "849116bf-fbe4-4dd0-9187-92817cee17fb" (UID: "849116bf-fbe4-4dd0-9187-92817cee17fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:38.243866 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.243839 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849116bf-fbe4-4dd0-9187-92817cee17fb-kube-api-access-sfrhm" pod "849116bf-fbe4-4dd0-9187-92817cee17fb" (UID: "849116bf-fbe4-4dd0-9187-92817cee17fb"). InnerVolumeSpecName "kube-api-access-sfrhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:57:38.335088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.335047 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:38.335088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.335084 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfrhm\" (UniqueName: \"kubernetes.io/projected/849116bf-fbe4-4dd0-9187-92817cee17fb-kube-api-access-sfrhm\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:38.335088 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.335096 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849116bf-fbe4-4dd0-9187-92817cee17fb-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:38.975083 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.975047 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" Nov 02 22:57:38.975286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.975081 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kxl2q" event={"ID":"849116bf-fbe4-4dd0-9187-92817cee17fb","Type":"ContainerDied","Data":"05001abda32a2476f5ad683e8297383e3dcf0a6c6bd5dd21d06f56ad03ee510a"} Nov 02 22:57:38.975286 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:38.975133 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05001abda32a2476f5ad683e8297383e3dcf0a6c6bd5dd21d06f56ad03ee510a" Nov 02 22:57:47.087049 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.087006 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2"] Nov 02 22:57:47.087499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.087462 5109 cpu_manager.go:414] "containerMap: removing container" podUID="849116bf-fbe4-4dd0-9187-92817cee17fb" containerName="revoke-keys" Nov 02 22:57:47.087499 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.087475 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849116bf-fbe4-4dd0-9187-92817cee17fb" containerName="revoke-keys" Nov 02 22:57:47.087607 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.087547 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="849116bf-fbe4-4dd0-9187-92817cee17fb" containerName="revoke-keys" Nov 02 22:57:47.090882 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.090865 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.146819 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.146775 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2"] Nov 02 22:57:47.194336 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.194304 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:57:47.194552 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.194352 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zf6xm\"" Nov 02 22:57:47.221225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.221190 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6mb\" (UniqueName: \"kubernetes.io/projected/9cb1a8ac-4b65-4ba9-b597-34ac52931098-kube-api-access-qj6mb\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.221225 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.221230 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.221512 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.221378 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.322013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.321975 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6mb\" (UniqueName: \"kubernetes.io/projected/9cb1a8ac-4b65-4ba9-b597-34ac52931098-kube-api-access-qj6mb\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.322013 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.322019 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.322271 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.322124 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.322844 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.322817 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.324651 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.324622 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.383655 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.383583 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6mb\" (UniqueName: \"kubernetes.io/projected/9cb1a8ac-4b65-4ba9-b597-34ac52931098-kube-api-access-qj6mb\") pod \"maas-api-revoke-sub-6b19338f-cbcn2\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.403351 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.403319 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:47.773654 ip-10-0-143-75 kubenswrapper[5109]: W1102 22:57:47.773621 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb1a8ac_4b65_4ba9_b597_34ac52931098.slice/crio-f81101b2d474e6212deb4a89c7afef9f0ea2ed1a67cfba9858a73b036d3ae616 WatchSource:0}: Error finding container f81101b2d474e6212deb4a89c7afef9f0ea2ed1a67cfba9858a73b036d3ae616: Status 404 returned error can't find the container with id f81101b2d474e6212deb4a89c7afef9f0ea2ed1a67cfba9858a73b036d3ae616 Nov 02 22:57:47.774098 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:47.773994 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2"] Nov 02 22:57:48.017729 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:48.017688 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" event={"ID":"9cb1a8ac-4b65-4ba9-b597-34ac52931098","Type":"ContainerStarted","Data":"f81101b2d474e6212deb4a89c7afef9f0ea2ed1a67cfba9858a73b036d3ae616"} Nov 02 22:57:49.023894 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:49.023858 5109 generic.go:356] "Generic (PLEG): container finished" podID="9cb1a8ac-4b65-4ba9-b597-34ac52931098" containerID="8dfe1c820e75e7e300155e7546447c58796d4236a976ee7f6397e5ead7cbad66" exitCode=0 Nov 02 22:57:49.024316 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:49.023943 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" event={"ID":"9cb1a8ac-4b65-4ba9-b597-34ac52931098","Type":"ContainerDied","Data":"8dfe1c820e75e7e300155e7546447c58796d4236a976ee7f6397e5ead7cbad66"} Nov 02 22:57:50.170165 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.170141 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:50.253489 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.253454 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-openshift-ca\") pod \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " Nov 02 22:57:50.253703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.253548 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb1a8ac-4b65-4ba9-b597-34ac52931098-kube-api-access-qj6mb\" (UniqueName: \"kubernetes.io/projected/9cb1a8ac-4b65-4ba9-b597-34ac52931098-kube-api-access-qj6mb\") pod \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " Nov 02 22:57:50.253703 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.253608 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-xks-ca\") pod \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\" (UID: \"9cb1a8ac-4b65-4ba9-b597-34ac52931098\") " Nov 02 22:57:50.253912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.253850 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-openshift-ca" pod "9cb1a8ac-4b65-4ba9-b597-34ac52931098" (UID: "9cb1a8ac-4b65-4ba9-b597-34ac52931098"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:57:50.254010 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.253993 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-openshift-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:50.255782 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.255761 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-xks-ca" pod "9cb1a8ac-4b65-4ba9-b597-34ac52931098" (UID: "9cb1a8ac-4b65-4ba9-b597-34ac52931098"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:50.255911 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.255889 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb1a8ac-4b65-4ba9-b597-34ac52931098-kube-api-access-qj6mb" pod "9cb1a8ac-4b65-4ba9-b597-34ac52931098" (UID: "9cb1a8ac-4b65-4ba9-b597-34ac52931098"). InnerVolumeSpecName "kube-api-access-qj6mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:57:50.355075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.354977 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6mb\" (UniqueName: \"kubernetes.io/projected/9cb1a8ac-4b65-4ba9-b597-34ac52931098-kube-api-access-qj6mb\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:50.355075 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:50.355019 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb1a8ac-4b65-4ba9-b597-34ac52931098-maas-api-xks-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:51.035050 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:51.035017 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" event={"ID":"9cb1a8ac-4b65-4ba9-b597-34ac52931098","Type":"ContainerDied","Data":"f81101b2d474e6212deb4a89c7afef9f0ea2ed1a67cfba9858a73b036d3ae616"} Nov 02 22:57:51.035244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:51.035058 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cbcn2" Nov 02 22:57:51.035244 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:57:51.035066 5109 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81101b2d474e6212deb4a89c7afef9f0ea2ed1a67cfba9858a73b036d3ae616" Nov 02 22:57:56.490548 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:57:56.490507 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:58:17.728735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:17.728698 5109 scope.go:122] "RemoveContainer" containerID="4e6955ad420f531de5eff062d115cc9142bb0fc457deb47ffe01705e977280eb" Nov 02 22:58:17.737852 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:17.737827 5109 scope.go:122] "RemoveContainer" containerID="b2fb247300ec32bb0b23441c57ad0a5995f115c422a340eb59e0f0d17d4abd27" Nov 02 22:58:20.369322 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:20.369281 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hbxxd_0e10c6a1-006b-42ce-b75e-be8f3ce0031d/manager/0.log" Nov 02 22:58:20.603005 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:20.602971 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p2rtg_43205cd0-50b8-4766-805b-25961e99fa03/manager/0.log" Nov 02 22:58:21.440548 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:21.440515 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-546bfbd88f-9cdcm_8783ce4a-aa7d-4a16-aa4f-7cd954884912/manager/0.log" Nov 02 22:58:21.805769 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:21.805741 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8tqw4_e3308a94-eb6a-47eb-9283-be6bfd598b5f/server/0.log" Nov 02 22:58:23.640827 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:23.640792 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kxl2q_849116bf-fbe4-4dd0-9187-92817cee17fb/revoke-keys/0.log" Nov 02 22:58:23.870435 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:23.870411 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cbcn2_9cb1a8ac-4b65-4ba9-b597-34ac52931098/revoke-keys/0.log" Nov 02 22:58:24.095578 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:24.095538 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zmtw8_217f9ab7-a5f6-471f-a8fa-d346c6717509/revoke-keys/0.log" Nov 02 22:58:24.318735 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:24.318699 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4bckh_69bc7953-cb2e-4743-98af-65a14cac0ec1/postgres/0.log" Nov 02 22:58:26.490363 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:58:26.490314 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:58:26.835201 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:26.835173 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-djhzw_59f64123-31a0-4e5a-bf58-d6232e269329/kuadrant-console-plugin/0.log" Nov 02 22:58:28.850122 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:28.850084 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mcdtp_9f20cd24-bbe4-417b-911d-937733fcf433/discovery/0.log" Nov 02 22:58:29.900038 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:29.899997 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-9jkwk_c7eafe82-f30e-429e-907e-0d60710314a8/istio-proxy/0.log" Nov 02 22:58:30.176456 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:30.176362 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bdbv7_7dea2654-d13d-484f-ad7a-982fea3bffec/istio-proxy/0.log" Nov 02 22:58:31.967601 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:31.967546 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-tk2q6_c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa/istio-proxy/0.log" Nov 02 22:58:32.471940 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:32.471906 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9vjjw_72f8a56d-ed2f-4fc9-a5f5-ab414e384512/payload-pre-processing/0.log" Nov 02 22:58:32.685608 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:32.685554 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vng6p_5ce52112-b66d-4857-921c-ffcde641d781/payload-processing/0.log" Nov 02 22:58:33.764177 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:33.764149 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8_2f8e0bdb-a3c5-45b7-8539-4454e9da8d23/main/0.log" Nov 02 22:58:34.296207 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:34.296176 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8hsg9_207b3952-7783-4159-ba11-835f76be56f4/main/0.log" Nov 02 22:58:34.729338 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:34.729257 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts_f2124f9b-a6c2-401d-99fd-8c0ecf601f26/main/0.log" Nov 02 22:58:35.190309 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:35.190273 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9_edb523e3-3fad-4570-9d85-3c7798aba687/main/0.log" Nov 02 22:58:56.036631 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:56.036531 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hbxxd_0e10c6a1-006b-42ce-b75e-be8f3ce0031d/manager/0.log" Nov 02 22:58:56.247813 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:56.247779 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p2rtg_43205cd0-50b8-4766-805b-25961e99fa03/manager/0.log" Nov 02 22:58:56.489826 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:58:56.489738 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:58:56.936292 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:56.936253 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-546bfbd88f-9cdcm_8783ce4a-aa7d-4a16-aa4f-7cd954884912/manager/0.log" Nov 02 22:58:57.234335 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:57.234241 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8tqw4_e3308a94-eb6a-47eb-9283-be6bfd598b5f/server/0.log" Nov 02 22:58:58.810609 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:58.810554 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kxl2q_849116bf-fbe4-4dd0-9187-92817cee17fb/revoke-keys/0.log" Nov 02 22:58:59.029202 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:59.029174 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cbcn2_9cb1a8ac-4b65-4ba9-b597-34ac52931098/revoke-keys/0.log" Nov 02 22:58:59.235906 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:59.235817 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zmtw8_217f9ab7-a5f6-471f-a8fa-d346c6717509/revoke-keys/0.log" Nov 02 22:58:59.448762 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:58:59.448728 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4bckh_69bc7953-cb2e-4743-98af-65a14cac0ec1/postgres/0.log" Nov 02 22:59:01.636740 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:01.636713 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-djhzw_59f64123-31a0-4e5a-bf58-d6232e269329/kuadrant-console-plugin/0.log" Nov 02 22:59:03.396804 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:03.396773 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mcdtp_9f20cd24-bbe4-417b-911d-937733fcf433/discovery/0.log" Nov 02 22:59:14.452299 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:14.452262 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-9jkwk_c7eafe82-f30e-429e-907e-0d60710314a8/istio-proxy/0.log" Nov 02 22:59:14.772491 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:14.772457 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bdbv7_7dea2654-d13d-484f-ad7a-982fea3bffec/istio-proxy/0.log" Nov 02 22:59:16.702660 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:16.702625 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-tk2q6_c0d8c50f-fd0e-46e1-8fb0-18e5fa0e51fa/istio-proxy/0.log" Nov 02 22:59:17.205542 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:17.205509 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9vjjw_72f8a56d-ed2f-4fc9-a5f5-ab414e384512/payload-pre-processing/0.log" Nov 02 22:59:17.423809 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:17.423771 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vng6p_5ce52112-b66d-4857-921c-ffcde641d781/payload-processing/0.log" Nov 02 22:59:17.762747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:17.762717 5109 scope.go:122] "RemoveContainer" containerID="8840d3455e63ff9cea41c9e2e84658667187e900adaa3c8ffcf3615535ce0e39" Nov 02 22:59:18.265912 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:18.265883 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-k8bk8_2f8e0bdb-a3c5-45b7-8539-4454e9da8d23/main/0.log" Nov 02 22:59:18.763747 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:18.763718 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8hsg9_207b3952-7783-4159-ba11-835f76be56f4/main/0.log" Nov 02 22:59:19.209413 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:19.209336 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jh7ts_f2124f9b-a6c2-401d-99fd-8c0ecf601f26/main/0.log" Nov 02 22:59:19.669842 ip-10-0-143-75 kubenswrapper[5109]: I1102 22:59:19.669809 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ctpg9_edb523e3-3fad-4570-9d85-3c7798aba687/main/0.log" Nov 02 22:59:26.490678 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:59:26.490637 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 22:59:56.490104 ip-10-0-143-75 kubenswrapper[5109]: E1102 22:59:56.490064 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 23:00:00.172382 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.172346 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4"] Nov 02 23:00:00.172830 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.172790 5109 cpu_manager.go:414] "containerMap: removing container" podUID="9cb1a8ac-4b65-4ba9-b597-34ac52931098" containerName="revoke-keys" Nov 02 23:00:00.172830 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.172802 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb1a8ac-4b65-4ba9-b597-34ac52931098" containerName="revoke-keys" Nov 02 23:00:00.172916 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.172888 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb1a8ac-4b65-4ba9-b597-34ac52931098" containerName="revoke-keys" Nov 02 23:00:00.176135 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.176117 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.239781 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.239749 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/915d667e-bd03-4720-938b-e3becd310270-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-b8ch4\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.239994 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.239805 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2z4\" (UniqueName: \"kubernetes.io/projected/915d667e-bd03-4720-938b-e3becd310270-kube-api-access-rc2z4\") pod \"maas-api-key-cleanup-29894340-b8ch4\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.272539 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.272503 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zf6xm\"" Nov 02 23:00:00.274349 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.274324 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4"] Nov 02 23:00:00.340888 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.340844 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/915d667e-bd03-4720-938b-e3becd310270-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-b8ch4\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.341089 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.340901 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2z4\" (UniqueName: \"kubernetes.io/projected/915d667e-bd03-4720-938b-e3becd310270-kube-api-access-rc2z4\") pod \"maas-api-key-cleanup-29894340-b8ch4\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.341503 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.341483 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/915d667e-bd03-4720-938b-e3becd310270-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-b8ch4\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.399355 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.399316 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2z4\" (UniqueName: \"kubernetes.io/projected/915d667e-bd03-4720-938b-e3becd310270-kube-api-access-rc2z4\") pod \"maas-api-key-cleanup-29894340-b8ch4\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.498769 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.498677 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:00.872512 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:00.872481 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4"] Nov 02 23:00:00.874768 ip-10-0-143-75 kubenswrapper[5109]: W1102 23:00:00.874733 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915d667e_bd03_4720_938b_e3becd310270.slice/crio-7f6d8684f2b61a59b4dd7e4b7bf2b8264f21574b3f14170cc7a7b8d07c6d5a9a WatchSource:0}: Error finding container 7f6d8684f2b61a59b4dd7e4b7bf2b8264f21574b3f14170cc7a7b8d07c6d5a9a: Status 404 returned error can't find the container with id 7f6d8684f2b61a59b4dd7e4b7bf2b8264f21574b3f14170cc7a7b8d07c6d5a9a Nov 02 23:00:01.691490 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:01.691447 5109 generic.go:356] "Generic (PLEG): container finished" podID="915d667e-bd03-4720-938b-e3becd310270" containerID="feefcaaad88b97e55ca06265c4a782519f6ba068259414d3492c80ea69afe034" exitCode=60 Nov 02 23:00:01.691955 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:01.691509 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" event={"ID":"915d667e-bd03-4720-938b-e3becd310270","Type":"ContainerDied","Data":"feefcaaad88b97e55ca06265c4a782519f6ba068259414d3492c80ea69afe034"} Nov 02 23:00:01.691955 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:01.691555 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" event={"ID":"915d667e-bd03-4720-938b-e3becd310270","Type":"ContainerStarted","Data":"7f6d8684f2b61a59b4dd7e4b7bf2b8264f21574b3f14170cc7a7b8d07c6d5a9a"} Nov 02 23:00:01.691955 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:01.691833 5109 scope.go:122] "RemoveContainer" containerID="feefcaaad88b97e55ca06265c4a782519f6ba068259414d3492c80ea69afe034" Nov 02 23:00:02.698112 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:02.698027 5109 generic.go:356] "Generic (PLEG): container finished" podID="915d667e-bd03-4720-938b-e3becd310270" containerID="54899ffaa8d81223d0786d91237f39fec0f1c58a41e742c07f14cc024fd68e23" exitCode=60 Nov 02 23:00:02.698112 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:02.698075 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" event={"ID":"915d667e-bd03-4720-938b-e3becd310270","Type":"ContainerDied","Data":"54899ffaa8d81223d0786d91237f39fec0f1c58a41e742c07f14cc024fd68e23"} Nov 02 23:00:02.698661 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:02.698135 5109 scope.go:122] "RemoveContainer" containerID="feefcaaad88b97e55ca06265c4a782519f6ba068259414d3492c80ea69afe034" Nov 02 23:00:02.698661 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:02.698447 5109 scope.go:122] "RemoveContainer" containerID="54899ffaa8d81223d0786d91237f39fec0f1c58a41e742c07f14cc024fd68e23" Nov 02 23:00:02.698744 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:00:02.698700 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-b8ch4_odh-ai-gateway-infra(915d667e-bd03-4720-938b-e3becd310270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" podUID="915d667e-bd03-4720-938b-e3becd310270" Nov 02 23:00:03.704978 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:03.704949 5109 scope.go:122] "RemoveContainer" containerID="54899ffaa8d81223d0786d91237f39fec0f1c58a41e742c07f14cc024fd68e23" Nov 02 23:00:03.705474 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:00:03.705250 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-b8ch4_odh-ai-gateway-infra(915d667e-bd03-4720-938b-e3becd310270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" podUID="915d667e-bd03-4720-938b-e3becd310270" Nov 02 23:00:11.259003 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:11.258962 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-546bfbd88f-9cdcm_8783ce4a-aa7d-4a16-aa4f-7cd954884912/manager/0.log" Nov 02 23:00:11.764004 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:11.763976 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-546bfbd88f-9cdcm_8783ce4a-aa7d-4a16-aa4f-7cd954884912/manager/0.log" Nov 02 23:00:12.548821 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:12.548781 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vng6p_5ce52112-b66d-4857-921c-ffcde641d781/payload-processing/0.log" Nov 02 23:00:12.778425 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:12.778392 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vng6p_5ce52112-b66d-4857-921c-ffcde641d781/payload-processing/0.log" Nov 02 23:00:15.885509 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:15.885477 5109 scope.go:122] "RemoveContainer" containerID="54899ffaa8d81223d0786d91237f39fec0f1c58a41e742c07f14cc024fd68e23" Nov 02 23:00:16.769708 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:16.769673 5109 generic.go:356] "Generic (PLEG): container finished" podID="915d667e-bd03-4720-938b-e3becd310270" containerID="fd2ddf0a828e25c76a2322b62bf0afee2d3c3a2e03299e963b5ed2459f34408e" exitCode=60 Nov 02 23:00:16.769895 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:16.769715 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" event={"ID":"915d667e-bd03-4720-938b-e3becd310270","Type":"ContainerDied","Data":"fd2ddf0a828e25c76a2322b62bf0afee2d3c3a2e03299e963b5ed2459f34408e"} Nov 02 23:00:16.769895 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:16.769770 5109 scope.go:122] "RemoveContainer" containerID="54899ffaa8d81223d0786d91237f39fec0f1c58a41e742c07f14cc024fd68e23" Nov 02 23:00:16.770187 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:16.770168 5109 scope.go:122] "RemoveContainer" containerID="fd2ddf0a828e25c76a2322b62bf0afee2d3c3a2e03299e963b5ed2459f34408e" Nov 02 23:00:16.770462 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:00:16.770439 5109 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-b8ch4_odh-ai-gateway-infra(915d667e-bd03-4720-938b-e3becd310270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" podUID="915d667e-bd03-4720-938b-e3becd310270" Nov 02 23:00:17.782955 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:17.782919 5109 scope.go:122] "RemoveContainer" containerID="a4a8675ba9a20c34007f77781fdfc44eb5ea20e61ab795a20fa8928c478d0488" Nov 02 23:00:17.891806 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:17.891767 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4"] Nov 02 23:00:18.033312 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.033258 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:18.115034 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.114995 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915d667e-bd03-4720-938b-e3becd310270-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/915d667e-bd03-4720-938b-e3becd310270-maas-api-service-ca\") pod \"915d667e-bd03-4720-938b-e3becd310270\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " Nov 02 23:00:18.115229 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.115113 5109 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915d667e-bd03-4720-938b-e3becd310270-kube-api-access-rc2z4\" (UniqueName: \"kubernetes.io/projected/915d667e-bd03-4720-938b-e3becd310270-kube-api-access-rc2z4\") pod \"915d667e-bd03-4720-938b-e3becd310270\" (UID: \"915d667e-bd03-4720-938b-e3becd310270\") " Nov 02 23:00:18.115364 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.115341 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915d667e-bd03-4720-938b-e3becd310270-maas-api-service-ca" pod "915d667e-bd03-4720-938b-e3becd310270" (UID: "915d667e-bd03-4720-938b-e3becd310270"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:00:18.117408 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.117387 5109 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915d667e-bd03-4720-938b-e3becd310270-kube-api-access-rc2z4" pod "915d667e-bd03-4720-938b-e3becd310270" (UID: "915d667e-bd03-4720-938b-e3becd310270"). InnerVolumeSpecName "kube-api-access-rc2z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:00:18.215921 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.215882 5109 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2z4\" (UniqueName: \"kubernetes.io/projected/915d667e-bd03-4720-938b-e3becd310270-kube-api-access-rc2z4\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:00:18.215921 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.215914 5109 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/915d667e-bd03-4720-938b-e3becd310270-maas-api-service-ca\") on node \"ip-10-0-143-75.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:00:18.782256 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.782225 5109 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" Nov 02 23:00:18.782473 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.782221 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4" event={"ID":"915d667e-bd03-4720-938b-e3becd310270","Type":"ContainerDied","Data":"7f6d8684f2b61a59b4dd7e4b7bf2b8264f21574b3f14170cc7a7b8d07c6d5a9a"} Nov 02 23:00:18.782473 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.782372 5109 scope.go:122] "RemoveContainer" containerID="fd2ddf0a828e25c76a2322b62bf0afee2d3c3a2e03299e963b5ed2459f34408e" Nov 02 23:00:18.939269 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.939230 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4"] Nov 02 23:00:18.985691 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:18.985649 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-b8ch4"] Nov 02 23:00:19.890937 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:00:19.890898 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915d667e-bd03-4720-938b-e3becd310270" path="/var/lib/kubelet/pods/915d667e-bd03-4720-938b-e3becd310270/volumes" Nov 02 23:00:26.491316 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:00:26.491281 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 23:00:56.490359 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:00:56.490311 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 23:01:26.491260 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:01:26.491220 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 23:01:31.910195 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:01:31.910156 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-546bfbd88f-9cdcm_8783ce4a-aa7d-4a16-aa4f-7cd954884912/manager/0.log" Nov 02 23:01:32.450355 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:01:32.450318 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-546bfbd88f-9cdcm_8783ce4a-aa7d-4a16-aa4f-7cd954884912/manager/0.log" Nov 02 23:01:43.242039 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:01:43.241957 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vng6p_5ce52112-b66d-4857-921c-ffcde641d781/payload-processing/0.log" Nov 02 23:01:43.460419 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:01:43.460392 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vng6p_5ce52112-b66d-4857-921c-ffcde641d781/payload-processing/0.log" Nov 02 23:01:53.244856 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:01:53.244820 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-knp2v_74da9552-2470-4a9f-8160-d3a29aa5d9f5/global-pull-secret-syncer/0.log" Nov 02 23:01:56.490397 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:01:56.490338 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor" Nov 02 23:01:56.659540 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:01:56.659511 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zzcqw_b5806a44-0c3e-4ea8-a72c-a8fe4a2aea7a/konnectivity-agent/0.log" Nov 02 23:01:57.849525 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:01:57.849491 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-75.us-west-2.compute.internal_52559cd016dd3e2df41639aa852bc053/haproxy/0.log" Nov 02 23:02:03.799431 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.799401 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-djhzw_59f64123-31a0-4e5a-bf58-d6232e269329/kuadrant-console-plugin/0.log" Nov 02 23:02:03.937960 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.937914 5109 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq"] Nov 02 23:02:03.938376 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.938351 5109 cpu_manager.go:414] "containerMap: removing container" podUID="915d667e-bd03-4720-938b-e3becd310270" containerName="cleanup" Nov 02 23:02:03.938376 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.938369 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915d667e-bd03-4720-938b-e3becd310270" containerName="cleanup" Nov 02 23:02:03.938376 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.938384 5109 cpu_manager.go:414] "containerMap: removing container" podUID="915d667e-bd03-4720-938b-e3becd310270" containerName="cleanup" Nov 02 23:02:03.938710 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.938392 5109 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915d667e-bd03-4720-938b-e3becd310270" containerName="cleanup" Nov 02 23:02:03.938710 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.938516 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="915d667e-bd03-4720-938b-e3becd310270" containerName="cleanup" Nov 02 23:02:03.938710 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.938536 5109 memory_manager.go:365] "RemoveStaleState removing state" podUID="915d667e-bd03-4720-938b-e3becd310270" containerName="cleanup" Nov 02 23:02:03.941810 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.941786 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:03.991051 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.991018 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-drfxk\"/\"kube-root-ca.crt\"" Nov 02 23:02:03.991051 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:03.991027 5109 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-drfxk\"/\"openshift-service-ca.crt\"" Nov 02 23:02:04.037593 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.037536 5109 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-drfxk\"/\"default-dockercfg-x5w7p\"" Nov 02 23:02:04.039640 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.039612 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq"] Nov 02 23:02:04.051783 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.051714 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-podres\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.051783 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.051747 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-proc\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.051951 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.051828 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tr5\" (UniqueName: \"kubernetes.io/projected/2b9456b3-67db-44d0-ae6f-31cc362099f2-kube-api-access-m5tr5\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.051951 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.051846 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-lib-modules\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.051951 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.051879 5109 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-sys\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153391 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153349 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-podres\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153391 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153395 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-proc\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153745 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153441 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tr5\" (UniqueName: \"kubernetes.io/projected/2b9456b3-67db-44d0-ae6f-31cc362099f2-kube-api-access-m5tr5\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153745 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153463 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-lib-modules\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153745 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153494 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-podres\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153745 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153510 5109 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-sys\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153745 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153536 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-proc\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153745 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153617 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-lib-modules\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.153745 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.153620 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b9456b3-67db-44d0-ae6f-31cc362099f2-sys\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.211288 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.211256 5109 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tr5\" (UniqueName: \"kubernetes.io/projected/2b9456b3-67db-44d0-ae6f-31cc362099f2-kube-api-access-m5tr5\") pod \"perf-node-gather-daemonset-z5tpq\" (UID: \"2b9456b3-67db-44d0-ae6f-31cc362099f2\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.254769 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.254727 5109 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:04.632729 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:04.632698 5109 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq"] Nov 02 23:02:04.634432 ip-10-0-143-75 kubenswrapper[5109]: W1102 23:02:04.634405 5109 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b9456b3_67db_44d0_ae6f_31cc362099f2.slice/crio-15d6df9d381684a983ca38b45376541e74884cf7038d4a4567d30a66035abc4c WatchSource:0}: Error finding container 15d6df9d381684a983ca38b45376541e74884cf7038d4a4567d30a66035abc4c: Status 404 returned error can't find the container with id 15d6df9d381684a983ca38b45376541e74884cf7038d4a4567d30a66035abc4c Nov 02 23:02:05.309191 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:05.309148 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" event={"ID":"2b9456b3-67db-44d0-ae6f-31cc362099f2","Type":"ContainerStarted","Data":"ff90050a3a3173964b018384553f78ed61d7d509ddb72ef32ee680f7f79605ff"} Nov 02 23:02:05.309703 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:05.309229 5109 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" event={"ID":"2b9456b3-67db-44d0-ae6f-31cc362099f2","Type":"ContainerStarted","Data":"15d6df9d381684a983ca38b45376541e74884cf7038d4a4567d30a66035abc4c"} Nov 02 23:02:05.309703 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:05.309363 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:05.418995 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:05.418939 5109 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" podStartSLOduration=2.418922837 podStartE2EDuration="2.418922837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 23:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:05.417560356 +0000 UTC m=+3361.682553587" watchObservedRunningTime="2026-11-02 23:02:05.418922837 +0000 UTC m=+3361.683916063" Nov 02 23:02:07.802640 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:07.802601 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce420e46-db02-4284-8adf-7db89e3fe8f0/alertmanager/0.log" Nov 02 23:02:08.215719 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:08.215618 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce420e46-db02-4284-8adf-7db89e3fe8f0/config-reloader/0.log" Nov 02 23:02:08.625577 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:08.625542 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce420e46-db02-4284-8adf-7db89e3fe8f0/kube-rbac-proxy-web/0.log" Nov 02 23:02:09.040701 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:09.040662 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce420e46-db02-4284-8adf-7db89e3fe8f0/kube-rbac-proxy/0.log" Nov 02 23:02:09.455982 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:09.455902 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce420e46-db02-4284-8adf-7db89e3fe8f0/kube-rbac-proxy-metric/0.log" Nov 02 23:02:09.882880 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:09.882850 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce420e46-db02-4284-8adf-7db89e3fe8f0/prom-label-proxy/0.log" Nov 02 23:02:10.045954 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:10.045917 5109 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8"] Nov 02 23:02:10.092092 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:10.092057 5109 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zmtw8"] Nov 02 23:02:10.295245 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:10.295213 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce420e46-db02-4284-8adf-7db89e3fe8f0/init-config-reloader/0.log" Nov 02 23:02:11.324453 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:11.324421 5109 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-z5tpq" Nov 02 23:02:11.890458 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:11.890416 5109 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="217f9ab7-a5f6-471f-a8fa-d346c6717509" path="/var/lib/kubelet/pods/217f9ab7-a5f6-471f-a8fa-d346c6717509/volumes" Nov 02 23:02:12.900481 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:12.900451 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-fzzfw_efae3ac8-4b9e-4598-99b0-33eb10cd1935/monitoring-plugin/0.log" Nov 02 23:02:14.608323 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:14.608293 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sslx_54880730-d3aa-4d76-8234-0dc766363113/node-exporter/0.log" Nov 02 23:02:15.025158 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:15.025133 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sslx_54880730-d3aa-4d76-8234-0dc766363113/kube-rbac-proxy/0.log" Nov 02 23:02:15.436362 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:15.436291 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sslx_54880730-d3aa-4d76-8234-0dc766363113/init-textfile/0.log" Nov 02 23:02:17.814915 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:17.814879 5109 scope.go:122] "RemoveContainer" containerID="37c765dac66f3fed6290fb053b5e721e60f38dcda50fa5ccd588a0d6aa9ea8d0" Nov 02 23:02:21.327732 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:21.327703 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rqwlc_28191529-9de8-4283-bd79-6b4de730919e/prometheus-operator/0.log" Nov 02 23:02:21.733997 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:21.733925 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rqwlc_28191529-9de8-4283-bd79-6b4de730919e/kube-rbac-proxy/0.log" Nov 02 23:02:22.582600 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:22.581714 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-674c5d5fcb-845r7_b1cc496e-e59e-4f60-87bd-eeb2b18a165c/telemeter-client/0.log" Nov 02 23:02:22.995258 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:22.995185 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-674c5d5fcb-845r7_b1cc496e-e59e-4f60-87bd-eeb2b18a165c/reload/0.log" Nov 02 23:02:23.404690 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:23.404648 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-674c5d5fcb-845r7_b1cc496e-e59e-4f60-87bd-eeb2b18a165c/kube-rbac-proxy/0.log" Nov 02 23:02:23.821002 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:23.820970 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf65b7c55-xx8wq_63b6f934-07bb-42c1-8e3d-5bc2369b1716/thanos-query/0.log" Nov 02 23:02:24.230042 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:24.229960 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf65b7c55-xx8wq_63b6f934-07bb-42c1-8e3d-5bc2369b1716/kube-rbac-proxy-web/0.log" Nov 02 23:02:24.641660 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:24.641634 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf65b7c55-xx8wq_63b6f934-07bb-42c1-8e3d-5bc2369b1716/kube-rbac-proxy/0.log" Nov 02 23:02:25.052485 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:25.052455 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf65b7c55-xx8wq_63b6f934-07bb-42c1-8e3d-5bc2369b1716/prom-label-proxy/0.log" Nov 02 23:02:25.463106 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:25.463033 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf65b7c55-xx8wq_63b6f934-07bb-42c1-8e3d-5bc2369b1716/kube-rbac-proxy-rules/0.log" Nov 02 23:02:25.878583 ip-10-0-143-75 kubenswrapper[5109]: I1102 23:02:25.878545 5109 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bf65b7c55-xx8wq_63b6f934-07bb-42c1-8e3d-5bc2369b1716/kube-rbac-proxy-metrics/0.log" Nov 02 23:02:26.490142 ip-10-0-143-75 kubenswrapper[5109]: E1102 23:02:26.490107 5109 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p27gn" containerName="extractor"